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I.  INTRODUCTION 


According  to  the  Federal  Acquisition  Regulation  (FAR),  there  is  an  obligation  for 
any  government  acquisition  team  to  “deliver  on  a  timely  basis  the  best  value  product  or 
service  to  the  customer,  while  maintaining  the  public’s  trust  and  fulfilling  public  policy 
objectives”  (FAR  1.102[a]).  Acquisition  professionals  are  given  significant  discretion  on 
exactly  how  they  will  meet  that  mandate.  As  acquisition  teams  develop  their  strategy,  one 
of  the  most  critical  aspects  to  consider  is  the  source  selection  methodology  that  will 
derive  the  best  value  outcome  for  the  government. 

The  FAR  outlines  a  range  of  source  selection  methodologies  in  the  best  value 
continuum  from  the  lowest  price  technically  acceptable  (LPTA)  method  at  one  end  to  the 
highest  technically  rated  offeror  (HTRO)  at  the  other  (FAR  15.101-1).  Because  the  FAR 
mandates  that  “price  or  cost  to  the  Government  shall  be  evaluated  in  every  source 
selection”  (FAR  1 5.304(c)[  1  ]),  the  tradeoff  method  is  utilized  most  often  in  lieu  of 
HTRO.  The  source  selection  methodology  to  be  utilized  to  achieve  best  value  depends  on 
the  complexity  of  the  requirement  as  well  as  the  importance  of  price  or  cost  and  non-cost 
factors  to  be  evaluated  (FAR  15.101).  According  to  Calisti  (2015),  “industry  contends, 
overusing  LPTA  in  the  long  haul  will  erode  the  DOD  technological  edge  through  low- 
cost/low-performance  solutions;  cause  perfonnance  innovators  to  depart  the  market  and 
reduce  the  quality  of  goods  and  services  provided”  (p.  17).  The  government  on  the  other 
hand  believes  that  if  LPTA  is  utilized  properly  by  ensuring  that  the  technical 
acceptability  is  at  the  appropriate  level,  it  can  achieve  best  value  more  efficiently 
(Kendall,  2012).  This  research  will  analyze  contracts  that  were  awarded  under  both 
LPTA  and  tradeoff  methods  to  determine  whether  a  relationship  exists  between 
performance  outcomes  and  the  source  selection  method  at  the  Air  Force  Materiel 
Command. 

A.  PROBLEM  STATEMENT 

There  is  an  anecdotal  belief  that  the  government  receives  a  higher  quality  product 
or  superior  level  of  perfonnance  from  source  selections  conducted  through  the  tradeoff 
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method  versus  LPTA.  However,  there  is  no  empirical  evidence  to  prove  that  this  belief  is 
true.  There  is  not  a  significant  body  of  research  to  validate  whether  contractor 
performance  outcomes  from  tradeoff  selection  methods  result  in  better  value  to  the 
government  and  that  outcomes  from  LPTA  methods  result  in  inferior  quality. 

B.  PURPOSE 

This  research  is  aimed  at  determining  whether  there  is  a  relationship  between  the 
source  selection  method  (LPTA  or  tradeoff)  and  the  subsequent  outcome  of  the  contract. 
Understanding  this  relationship  will  help  the  Air  Force  better  prepare  for  future 
acquisitions.  The  ability  to  select  the  proper  source  selection  method  for  each  acquisition 
scenario  will  ensure  that  the  right  amount  of  resources  get  committed  to  each  project 
which  reduces  waste  in  the  acquisition  process. 

C.  RESEARCH  OBJECTIVES  AND  QUESTIONS 

The  objective  of  this  research  is  to  analyze  whether  a  relationship  exists  between 
source  selection  methods  (LPTA  or  tradeoff)  and  the  level  of  resulting  perfonnance 
outcomes.  The  contractor’s  performance  results  shall  be  obtained  from  the  Contractor 
Performance  Assessment  Reporting  System  (CPARS)  and  Earned  Value  Management 
System  (EVMS).  The  following  questions  form  the  basis  for  this  research: 

1 .  What  is  the  relationship  between  a  particular  source  selection  method  and 
the  resulting  performance  outcome  of  the  contractor? 

2.  What  is  the  relationship  between  a  particular  source  selection  method  and 
the  PALT? 

i.  If  there  is  a  positive  relationship,  can  the  amount  of  additional 
resources  utilized  for  that  particular  source  selection  method  be 
quantified  in  terms  of  man-hours  or  in  dollars? 

ii.  Does  the  resulting  performance  outcome  merit  the  use  of  that 
particular  source  selection  method? 

D.  METHODOLOGY 

The  methodology  for  this  research  include  a  literature  review  of  certain  aspects  of 
the  federal  contracting  process  which  impacts  decisions  on  source  selection  methods,  the 
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best  value  continuum,  source  selection  procedures,  controversial  issues  over  the  use  of 
LPTA,  and  the  need  for  source  selection  training  for  federal  acquisition  professionals. 
Data  collection  was  limited  to  contracts  awarded  within  the  Air  Force  Life  Cycle 
Management  Center  (AFLCMC)  Directorates  located  at  Wright-Patterson  AFB  in 
Dayton,  OH.  The  contracts  reviewed  for  this  research  were  selected  based  on  an  award 
value  in  excess  of  $  1  million  which  is  the  minimum  threshold  for  CPARS  reporting,  an 
award  date  between  fiscal  years  2010  and  2013,  and  competition  using  either  LPTA  or 
tradeoff  source  selection  methods.  The  Federal  Procurement  Data  System-Next 
Generation  (FPDS-NG)  was  used  to  identify  a  list  of  contracts  awarded  between  2010 
and  2013.  The  AFLCMC  Directorates  were  then  asked  to  identify  and  physically  provide 
all  competitively  awarded  contracts  from  the  list  to  include  source  selection  material  to 
be  analyzed  by  the  research  team. 

E.  BENEFITS 

This  research  seeks  to  identify  whether  a  particular  source  selection  method 
produces  better  contractor  performance  outcomes  than  another  as  well  as  to  quantify  the 
government  resources  needed  to  conduct  that  particular  source  selection  method.  The 
results  of  this  research  will  provide  federal  acquisition  professionals  with  better  insight 
on  what  outcomes  to  expect  as  well  as  the  level  of  resources  required  to  pursue  these 
source  selection  methods.  These  insights  will  help  in  future  acquisition  planning  to 
reduce  waste,  streamline  the  acquisition  process,  and  achieve  best  value. 

F.  LIMITATIONS 

There  are  many  limitations  to  this  research  which  impacts  the  applicability  of  the 
results  across  the  DOD.  First,  the  data  is  only  limited  to  Air  Force  contracts  from  one 
particular  Major  Command  (MAJCOM)  and  only  one  contracting  center  within  that 
MAJCOM.  Moreover,  due  to  time  and  manpower  constraints,  it  was  impossible  to  collect 
a  significant  amount  of  data.  Finally,  the  data  collected  only  contains  contracts  awarded 
from  2010  to  2013,  which  leaves  a  vast  number  of  contracts  that  would  otherwise  qualify 
for  this  study  unobserved.  As  more  studies  of  this  nature  are  conducted  across  other 
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installations  and  agencies,  the  findings  will  have  even  greater  significance  for 
government  acquisition  professionals  across  the  DOD. 


G.  ORGANIZATION  OF  THE  REPORT 

This  report  is  organized  into  five  chapters  to  include  this  introduction.  The  next 
chapter  (Chapter  II)  provides  a  literature  review  on  the  federal  contracting  process,  the 
best  value  continuum,  source  selection  procedures,  and  a  discussion  of  the  controversy 
over  increased  utilization  of  the  LPTA  source  selection  method  in  the  DOD.  Chapter  III 
discusses  the  methodology  for  this  research  to  include  background  infonnation  regarding 
the  AFLCMC,  the  methods  used  to  identify  the  data,  and  how  the  data  was  obtained. 
Chapter  IV  outlines  the  results  and  findings  of  the  data  analysis  as  well  as  answers  the 
research  questions.  The  final  chapter  (Chapter  V)  summarizes  and  concludes  this  research 
report  with  a  recommendation  for  future  research. 

H.  SUMMARY 

This  chapter  examined  the  background  infonnation  on  the  basis  of  this  research  to 
include  the  problem  statement,  purpose,  research  objectives  and  questions,  the 
methodology,  as  well  as  the  benefits  and  limitations  of  this  study.  The  next  chapter  is  the 
literature  review  which  provides  an  overview  of  statutes  important  to  the  government 
contracting  process.  Additionally,  it  presents  the  best  value  continuum,  highlighting  the 
LPTA  and  tradeoff  source  selection  methods.  Finally,  the  literature  review  examines  the 
source  selection  procedures  and  discusses  the  controversial  issues  over  the  increased  use 
of  LPTA  in  the  DOD. 
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II.  LITERATURE  REVIEW 


The  purpose  of  this  chapter  is  to  provide  an  overview  of  general  information  on 
the  federal  contracting  process,  which  influences  decisions  about  the  source  selection 
methods  to  be  used  in  federal  procurement.  Specifically,  this  chapter  begins  with  a 
review  of  applicable  statutes  that  guide  the  management  of  federal  contracts  and  mandate 
competition  as  well  as  efficiency  in  the  federal  procurement  process.  Further,  the  chapter 
reviews  the  various  source  selection  methods  available  in  the  best-value  continuum, 
highlighting  the  important  differences  between  LPT  A  and  tradeoff.  Additionally,  this 
chapter  examines  the  DOD  source  selection  procedures,  as  well  as  efforts  made  to  reform 
these  procedures  to  maximize  the  concept  of  best  value.  The  chapter  concludes  with  a 
discussion  of  a  number  of  controversial  issues  surrounding  the  increased  use  of  LPT  A 
and  the  need  for  robust  source  selection  training  for  federal  acquisition  professionals. 

A.  CONTRACT  MANAGEMENT  STATUTES  AND  REGULATIONS 

The  regulatory  agency  responsible  for  “shaping  the  policies  and  practices  federal 
agencies  use  to  acquire  the  goods  and  services  they  need  to  carry  out  their 
responsibilities”  (White  House,  n.d.,  OMB  section  para.  2)  to  the  tune  of  approximately 
$530  billion  each  year  is  called  the  Office  of  Federal  Procurement  Policy  (OFPP).  The 
OFPP  was  established  by  Congress  to  “provide  overall  direction  for  government-wide 
procurement  policies,  regulations,  and  procedures  and  to  promote  economy,  efficiency, 
and  effectiveness  in  acquisition  processes”  (White  House,  n.d.,  OMB  section  para.  2). 
Important  statutes  provide  the  foundation  for  critical  aspects  of  government  contracting, 
particularly  decisions  about  source  selection  methods  to  be  utilized.  These  include  the 
Competition  in  Contracting  Act  (CICA)  and  the  Federal  Acquisition  Streamlining  Act 
(FASA). 

1.  Competition  in  Contracting  Act 

The  mandate  for  competition  is  the  key  factor  that  forces  agencies  to  conduct 

source  selections  for  procurement  actions.  Competition  not  only  helps  the  government 

achieve  best  value  at  a  fair  and  reasonable  price,  but  it  is  also  vital  to  the  small  business 
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program,  the  defense  industry,  and  the  economy.  The  CICA,  enacted  in  1984,  requires 
that  all  contemplated  actions  leading  to  the  award  of  contracts  must  be  entered  into  a  full 
and  open  competition  (Manuel,  2011).  FAR  6. 101  prescribes  that 

contracting  officers  shall  provide  for  full  and  open  competition  through 
use  of  the  competitive  procedures(s)  contained  in  this  subpart  that  are  best 
suited  to  the  circumstances  of  the  contract  action  and  consistent  with  the 
need  to  fulfill  the  government’s  requirements  efficiently.  (FAR  6.101  [b]) 

While  the  promotion  of  maximum  competition  is  preferred,  there  are 
circumstances  that  prevent  the  government  from  being  able  to  compete  certain  types  of 
requirements  or  when  a  sole  source  environment  is  more  advantageous  to  the 
government.  In  such  cases,  the  FAR  “prescribes  policies  and  procedures,  and  identifies 
the  statutory  authorities,  for  contracting  without  providing  for  full  and  open  competition” 
(FAR  6.3).  The  source  selection  methods  and  procedures  for  competitive  awards  are 
discussed  in  more  detail  later  in  this  chapter. 

2.  Federal  Acquisition  Streamlining  Act 

Another  notable  statute  that  influences  the  decision  making  process  for  source 
selection  methods  by  mandating  efficiencies  in  the  federal  procurement  practices  is  the 
FASA.  Due  to  the  size  and  complexity  of  the  federal  acquisition  system,  Congress 
enacted  the  Federal  Acquisition  Streamlining  Act  of  1994  with  the  goal  of  achieving 
efficiencies  in  the  acquisition  process  (Drelicharz,  1994).  According  to  Drelicharz 
(1994),  the  intent  of  FASA  “is  to  develop  a  more  equitable  balance  between  government- 
unique  requirements  and  the  need  to  lower  the  government’s  cost  of  doing  business” 
(p.  12).  Drelicharz  (1994)  also  reports  that 

the  act  addresses  these  objectives  through  an  emphasis  on  increasing 
government-wide  reliance  on  the  use  of  commercial  practices,  goods  and 
services;  streamlining  the  rules  and  regulations  that  govern  high-volume 
contracting  activities,  which  represent  an  overall  low-dollar  expenditure; 
and  increasing  the  selling  opportunities  for  small  businesses,  (p.  12) 

Drelicharz  (1994)  explained  a  few  of  the  noteworthy  changes  brought  about  by 
the  enactment  of  the  FASA  include  the  following: 
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•  increasing  the  simplified  acquisition  threshold  (SAT)  from  $25,000  to 

$100,000; 

•  enabling  government  purchase  of  commercial  products  on  commercial 
terms; 

•  standardizing  the  procurement  system  throughout  all  federal  agencies; 
providing  notices  and  debriefs  to  unsuccessful  bidders  upon  request; 

•  providing  authority  to  the  General  Services  Board  of  Contract  Appeals 
(GSBCA)  to  “dismiss  frivolous  protests”  and  “invoke  procedural  sanctions 
where  a  person  makes  a  bad-faith  protests”  (Drelicharz,  1994,  P.15);  and 

•  requiring  the  establishment  of  a  Federal  Acquisition  Computer  Network 
(FACNET)  as  a  way  to  “dramatically  streamline  the  federal  purchasing 
process”  (Drelicharz,  1994,  p.  15). 

The  change  that  directly  impacts  source  selection  methods  is  the  new  SAT.  The 
increased  SAT  allows  contracting  officers  to  use  simplified  procedures  for  competitive 
awards  up  to  $100,000  under  FAR  13  in  lieu  of  formal  source  selection  procedures  under 
FAR  15  (Barry,  1995).  These  statutes  not  only  impact  decisions  pertaining  to  source 
selection  methods  but  also  affect  how  the  contract  management  processes  are  performed. 

B.  CONTRACT  MANAGEMENT  PROCESS 

The  contract  management  process  is  an  overarching  endeavor  that  includes  a 
number  of  activities  to  include  developing  a  requirement,  awarding  the  contract,  and 
managing  relationships  with  the  contractor  after  award.  In  his  book  titled  World  Class 
Contracting,  Garrett  defines  the  contract  management  process  as  “the  art  and  science  of 
managing  a  contractual  agreement  throughout  the  contracting  process”  (Garrett,  2015, 
p.  15).  Understanding  this  process  provides  an  important  foundation  to  the  factors  that 
impact  competition,  source  selection,  and  best  value. 

According  to  Garrett  (2015),  there  are  three  phases  and  six  primary  steps  in  the 
contract  management  process,  as  shown  in  Figure  1. 
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Figure  1 .  Contract  Management  Process 


Adapted  from:  Garrett,  G.  A.  (2015).  World  class  contracting  (6th  ed.).  Chicago,  IL: 
Wolters  Kluwer. 


For  purposes  of  this  research,  we  will  only  focus  on  the  processes  of  the  buyer 
since  we  will  be  analyzing  Air  Force  procurement  data  as  the  buying  agency.  The 
following  paragraphs  will  briefly  describe  the  three  phases  along  with  the  purpose  of 
each  step  and  the  activities  generally  performed  during  those  steps. 

1.  Pre-award  Phase 

According  to  Garrett  (2015),  the  pre-award  phase  is  mostly  concerned  with 
planning  for  the  acquisition,  determining  what  the  government  needs,  what  is  available  in 
the  market,  making  a  decision  to  provide  solutions  in-house  or  outsourcing,  and  soliciting 
the  market  to  acquire  the  needs.  Garrett  (2015)  identifies  procurement  planning, 
solicitation  planning,  and  solicitation  as  the  three  important  steps  to  focus  on  during  this 
phase  because  “this  phase  is  vital  in  creating  successful  business  relationships”  (Garrett, 
2015,  p.  17).  Since  using  contracts  to  procure  needs  involve  inherent  risks  associated  with 
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expectation  management  of  both  contracting  parties,  it  is  important  to  perform  each  step 
of  the  pre-award  phase  with  due  diligence  to  minimize  this  risk  (Garrett,  2015). 

a.  Step  1:  Procurement  Planning 

The  most  important  decision  to  be  made  during  the  procurement  planning  step 
according  to  Garrett  (2015)  is  the  “make-or-buy  decision  or  outsourcing  decision” 
(Garrett,  2015,  p.  20).  The  primary  focus  of  this  step  is  to  determine  what  the  needs  of  the 
organization  are,  when  it  must  be  filled,  and  how  (Garrett,  2015).  The  FAR  defines 
acquisition  planning  as  “the  process  by  which  the  efforts  of  all  personnel  responsible  for 
an  acquisition  are  coordinated  and  integrated  through  a  comprehensive  plan  for  fulfilling 
the  agency  need  in  a  timely  manner  and  at  a  reasonable  cost”  (FAR  2.101).  Some  of  the 
activities  performed  during  this  step  include  defining  the  needs  of  the  organization, 
analyzing  internal  resources  and  constraints,  market  intelligence,  and  risk  analysis 
(Garrett,  2015). 

Market  intelligence  and  market  research  are  critical  aspects  of  procurement 
planning  for  many  reasons.  For  one  thing,  they  help  identify  whether  the  requirement  is  a 
commercial  need.  Additionally,  they  identify  potential  companies  interested  in  the 
requirement  which  is  necessary  to  make  decisions  about  small  business  and  or 
socioeconomic  set-asides.  This  information  is  also  necessary  to  determine  the 
procurement  method  as  well  as  the  source  selection  method  such  as  LPTA  or  tradeoff. 
Hosting  industry  days  or  posting  sources  sought  notices  on  the  Government  Point  of 
Entry  (GPE)  are  great  ways  to  obtain  industry  information. 

Risk  analysis  is  another  important  activity  to  be  conducted  during  the 
procurement  planning  step  (Garrett,  2015).  In  order  to  minimize  the  various  types  of  risks 
associated  with  contracting,  “the  buyer  must  identify  the  sources  of  uncertainty  about 
contract  performance  and  the  risks  associated  with  those  uncertainties  to  write  a  contract 
that  will  address  those  risks  and  fairly  distribute  them  between  buyer  and  seller”  (Garrett, 
2015,  p.  73).  Risk  analysis  is  a  critical  activity  in  the  detennination  of  the  contract  type. 
Cost,  schedule,  and  performance  are  just  a  few  examples  of  risk  areas  that  should  be 
analyzed.  There  are  various  methods  to  avoid,  reduce,  or  transfer  risk;  however,  there  are 
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also  tradeoffs  or  costs  associated  with  them  as  well.  For  example,  cost  risk  could  be 
transferred  to  the  contractor  by  using  a  firm  fixed  price  type  of  contract.  In  turn,  the 
contractor  will  build  the  cost  of  his  or  her  risk  into  the  offering  price. 

The  personnel  involved  in  performing  such  activities  for  procurement  planning 
should  consist  of  a  multi-functional  team.  In  government  acquisitions,  a  typical 
acquisition  team  includes  members  from  the  requiring  organization,  contracting,  legal, 
and  finance.  The  results  of  these  activities  should  be  used  to  support  the  critical  decision 
of  sourcing  within  the  organization  or  acquiring  the  needs  from  outside  (Garrett,  2015). 
According  to  Garrett  (2015),  this  make  or  buy  decision  is  extremely  important  because 
the  decisions  can  impact  an  organization’s  strategy  and  may  be  critical.  The  activities 
conducted  during  the  procurement  planning  process  provide  for  an  important  foundation 
for  the  next  step  in  the  preaward  phase. 

b.  Step  2:  Solicitation  Planning 

With  a  fully  developed  description  of  what  is  required  and  the  decision  to  procure, 
the  next  step  in  the  pre-award  phase  is  solicitation  planning.  The  purpose  of  this  step  is  to 
develop  the  documentation  necessary  to  solicit  the  requirement  (Garrett,  2015). 
Depending  on  the  type  of  requirement,  these  documents  will  describe  what  the 
government  needs  and  how  the  work  should  be  performed  or  delivered.  A  draft 
solicitation  or  request  for  proposal  (RFP)  outlines  the  quantity  and  description  of  the 
items  or  services  to  be  performed,  the  contract  specifications  such  as  a  statement  of  work 
(SOW),  applicable  clauses,  instructions  on  how  offerors  should  respond,  and  how 
proposals  will  be  evaluated.  The  criteria  for  evaluating  proposals  may  vary  depending  on 
the  procurement  method  selected  as  well  as  the  source  selection  method  to  be  utilized. 

c.  Step  3:  Solicitation 

With  the  proper  documents  such  as  the  SOW  and  the  RFP  built,  the  next  step  in 

the  pre-award  phase  is  the  solicitation.  The  purpose  of  this  step  “consists  of  obtaining 

information  (bids  and  proposals)  from  prospective  sellers  on  how  project  needs  can  be 

met”  (Garrett,  2015,  p.  78).  During  solicitation,  a  pre-bid  conference  may  be  held  and  the 

requirement  advertised,  among  other  activities  (Garrett,  2015).  A  pre-bid  conference  can 

10 


be  used  to  ensure  that  interested  vendors  fully  understand  the  requirement  from  both  a 
technical  as  well  as  contractual  stand  point,  and  to  provide  clarifications  or  answer 
questions  which  can  lead  to  amendments  to  the  RFP  (Garrett,  2015).  In  a  competitive 
solicitation,  an  RFP  is  commonly  advertised  through  the  GPE  at  www.fbo.gov.  The  pre¬ 
award  phase  ends  with  the  receipt  of  proposals  or  bids  submitted  by  vendors  on  a  timely 
basis  as  a  result  of  the  solicitation  process  (Garrett,  2015). 

2.  Award  Phase 

With  all  the  proposals  in  house  and  ready  for  evaluations,  the  next  phase  of  the 
contract  management  process,  the  award  phase  is  ready  to  begin.  Source  selection  is  the 
only  step  of  this  phase  where  many  critical  activities  will  be  performed.  These  activities 
are  vital  to  the  successful  fulfillment  of  agency  needs  through  a  contract  (Garrett,  2015). 
The  government’s  goal  during  this  phase  is  to  make  an  award  to  an  offeror  that  can  meet 
the  buyer’s  needs  at  the  best  value.  The  DOD  Source  Selection  Procedures  are  the  most 
utilized  approach  to  making  such  awards  in  government  acquisitions. 

a.  Step  4:  Source  Selection 

Source  Selection  is  the  process  of  evaluating  the  proposals  received  in  order  to 
select  the  right  offeror  to  fill  the  needs.  For  competitive  awards,  the  methods  generally 
used  to  make  selections  are  LPTA  or  tradeoff  depending  on  how  well  the  requirement  is 
defined  (FAR  15.101).  According  to  Garrett  (2015),  “the  process  used  to  accomplish  this 
varies  dramatically  depending  on  the  company,  the  products  or  services  involved,  the 
complexity  of  the  procurement,  and  many  other  factors”  (p.  138).  The  key  to  evaluating 
proposals  successfully  in  government  acquisitions  is  to  ensure  that  the  evaluation  factors 
presented  in  Section  M  of  the  solicitation  is  followed  exactly  as  prescribed.  A  few  of  the 
commonly  evaluated  factors  in  government  requirements  are  price  or  cost,  technical 
approach,  and  past  performance.  Tools  such  as  contract  negotiation,  weighting  systems, 
screening  systems,  and  independent  estimates  may  be  used  to  evaluate  these  factors 
during  the  source  selection  process  to  achieve  the  optimum  outcome,  which  is  a  best 
value  contract  (Garrett,  2015). 
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3. 


Post  Award  Phase 


An  established  contract  with  a  selected  vendor  ready  to  perform  marks  the 
beginning  of  the  post-award  phase  of  the  contract  management  process.  This  final  phase 
involves  two  steps  to  ensure  that  ah  terms  and  conditions  of  the  contract  are  carried  out 
properly  and  that  the  contract  gets  closure  upon  performance  completion  or  termination 
(Garrett,  2015).  Ah  parties  involved  in  the  contract  will  need  to  stay  in  constant 
communication  in  order  to  achieve  the  desired  outcomes  of  the  contract  through 
performance,  progress,  and  contract  compliance  monitoring  (Garrett,  2015). 

a.  Step  5:  Contract  Administration 

Contract  administration,  according  to  Garrett  (2015),  “is  the  process  of  ensuring 
that  each  party’s  perfonnance  meets  contractual  requirements”  (p.  140).  Success  depends 
on  full  comprehension  of  the  requirements  and  responsibilities  of  both  parties;  therefore, 
the  contract  should  be  carefully  studied  by  ah  affected  personnel  and  organizations  on 
both  sides  (Garrett,  2015).  The  contract  administration  process  involves  many  activities 
with  a  variety  of  tools  and  techniques  to  help  perform  these  activities  effectively.  In  their 
book  titled  Management  of  Defense  Acquisition  Projects,  Rendon  and  Snider  (2008)  offer 
four  techniques  typically  utilized  in  the  government  contract  administration  process 
which  includes: 

1 .  Conducting  a  pre-performance  conference, 

2.  Monitoring  the  contractor’s  work  results, 

3.  Measuring  contractors  performance,  and 

4.  Managing  the  change-control  process  (Rendon  &  Snider,  2008,  p.  176). 

These  techniques  are  designed  to  ensure  that  ah  parties  are  standing  on  common 

ground  as  it  relates  to  the  requirements,  responsibilities,  processes,  and  protocols  that 
both  parties  will  follow  to  resolve  any  conflicts  or  disputes  until  the  contract  is  ready  for 
closeout. 


12 


b.  Step  6:  Contract  Closeout 

The  sixth  and  final  step  of  the  contract  management  process  deals  with  closeout 
or  termination.  According  to  Garrett  (2015),  “a  contract  can  end  in  one  of  three  ways: 
successful  performance,  mutual  agreement,  or  breach  of  contract”  (p.  159).  Regardless  of 
whether  a  contract  ends  due  to  successful  completion  or  tennination,  the  closeout  process 
must  ensure  that  all  administrative  matters  such  as  final  payments  or  settlements  are  paid 
and  that  documents  necessary  to  closeout  the  contract  are  received  (Garrett,  2015).  In 
government  acquisitions,  FAR  4.804  outlines  the  responsibilities  for  carrying  out  the 
closeout  process  which  primarily  focuses  on  the  acceptance  of  the  good  or  service, 
payment  to  the  contractor,  and  the  statutory  timelines  for  conducting  the  closeout. 

The  contract  management  process  shows  the  important  roles  each  of  the  phases 
and  steps  play  in  government  acquisitions.  Most  importantly,  this  process  clearly  shows 
that  best  value  cannot  be  achieved  through  just  one  single  phase  or  step.  The  extensive 
leg  work  performed  prior  to  and  after  the  source  selection  phase  of  the  entire  contract 
management  process  influences  the  objectives  of  achieving  best  value.  The  next  section 
provides  an  overview  of  the  best  value  concept  and  the  best  value  continuum  as 
prescribed  by  the  FAR. 

C.  BEST  VALUE  AND  BEST  VALUE  CONTINUUM 

The  underlying  goal  of  every  contract  in  the  federal  government  should  be  to 
acquire  a  product  or  service  that  provides  the  “best  value.”  The  FAR  defines  best  value  as 
an  “expected  outcome  of  an  acquisition  that,  in  the  government’s  estimation,  provides  the 
greatest  overall  benefit  in  response  to  the  requirement”  (FAR  2.101).  It  is  important  to 
note  that  best  value  is  not  always  achieved  by  awarding  to  the  lowest  priced  bidder. 
Certainly,  there  are  instances  where  price  should  be,  and  is,  the  overarching  factor  that 
determines  the  winning  bid.  Conversely,  there  are  times  when  factors  other  than  price  or 
cost,  such  as  a  technical  approach  to  complex  requirements  or  exceptional  past 
performance  records,  garner  additional  consideration  by  the  acquisition  team  as  a  way  to 
obtain  the  best  product  or  service. 
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In  order  to  address  the  myriad  degrees  of  complexity  in  government  requirements, 
the  FAR  lays  out  distinct  methods  by  which  best  value  can  be  achieved  through  the  use  of 
the  best  value  continuum.  The  interaction  of  price  and  non-cost  factors  that  affect  the 
source  selection  methods  to  be  utilized  according  to  the  best  value  continuum  is  clearly 
described  in  the  FAR: 

An  agency  can  obtain  best  value  in  negotiated  acquisitions  by  using  any 
one  or  a  combination  of  source  selection  approaches.  In  different  types  of 
acquisitions,  the  relative  importance  of  cost  or  price  may  vary.  For 
example,  in  acquisitions  where  the  requirement  is  clearly  definable  and  the 
risk  of  unsuccessful  contract  performance  is  minimal,  cost  or  price  may 
play  a  dominant  role  in  source  selection.  The  less  definitive  the 
requirement,  the  more  development  work  required,  or  the  greater  the 
performance  risk,  the  more  technical  or  past  performance  considerations 
may  play  a  dominant  role  in  source  selection.  (FAR  15.101) 

According  to  the  best  value  continuum,  as  seen  in  Figure  2,  the  relative 
importance  of  price  as  compared  to  non-cost  factors  determines  whether  the  LPTA  or 
tradeoff  method  should  be  used. 


Figure  2.  The  Best  Value  Continuum 


Across  the  Best  value  Continuum,  the  relative  importance  of  cost  or  price  may  vary. 


FU  15.101 


Tradeoff  Process 


Sum  of  Non- $  factors  Significantly  <  $ 


LPTA  is  used  at  the  endpoint  of  the 
Best  Value  Continuum  where  cost 
or  price  has  maximum  weight  and 
all  other  factors  are  pass/fail. 


Sum  of  Non-  $  foe  tors  approx.  -  $ 


Sum  of  Non-$  factors  Significantly  >  $ 


Per  statute,  quality  of  product  or  service  shall  be  oddressed  in  every  source  selection. isjo* 


Factors  Other  Than  Cost  or  Price 


Cost  or  Price  Factor 


DoO  Source  Selection  Procedures  Section 
2.3. 1.2  describes  the  evaluation  of  quality 
of  product  or  service  via  the  factors  of: 

•  Technical 

•  Past  Performance 


all  be  evaluated  in  every  source  selection.  fm  iiJ 


Best  Value  Continuum 


Adapted  from:  Best  value  continuum,  (n.d.).  In  ACQuipedia.  Retrieved  from 
https://dap.dau.mil/acquipedia/Pages/ArticleDetails.aspx7akR5201f734-3bce-4c5f-a5a3- 

47551df77ea5 
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The  LPT  A  method  is  at  one  end  of  the  continuum  with  the  heaviest  emphasis  on 
price  or  cost  as  the  most  important  factor.  The  tradeoff  method  is  a  process  that  moves 
away  from  the  LPTA  method  on  the  continuum  as  the  sum  of  non-cost  factors  such  as 
technical,  past  performance,  and  or  small  business  plan  becomes  equally  important  or 
more  important  than  price  or  cost. 

1.  Lowest  Price  Technically  Acceptable 

The  ideal  situation  for  using  the  LPTA  method  is  for  those  requirements  that  are 
well  defined  and  where  “the  solicitation  provides  the  evaluation  factors  and  significant 
subfactors  that  establish  the  requirements  of  acceptability”  (Rendon  &  Snider,  2008, 
p.  174).  One  aspect  that  makes  the  use  of  LPTA  desirable  over  tradeoff  is  the  ease  with 
which  technical  evaluations  can  be  conducted.  Since  all  technical  factors  are  based  on  an 
“acceptable”  or  “unacceptable”  basis,  the  award  is  simply  made  to  the  bidder  offering  the 
lowest  price  with  an  acceptable  rating  on  all  technical  factors. 

The  FAR  states  that  an  LPTA  approach  is  acceptable  only  when  the  government 
believes  that  “best  value  is  expected  to  result  from  selection  of  the  technically  acceptable 
proposal  with  the  lowest  evaluated  price”  (FAR  15.101-2).  Considering  this  definition, 
the  LPTA  method  is  best  suited  for  situations  where  the  government  would  not  realize 
any  additional  value  by  paying  higher  prices  for  non-cost  factors  that  exceed  minimum 
needs  (OUSD[AT&L],  2011).  This  is  primarily  the  case  in  acquisitions  for  non-complex 
or  commercially  available  products  or  services. 

2.  Tradeoff  Process 

When  contemplating  complex  acquisitions  with  a  degree  of  risk  that  can  impact 
cost,  schedule,  or  performance,  which  is  deemed  harmful  to  the  program,  the  best  source 
selection  method  is  the  tradeoff.  While  this  is  considered  a  more  resource-intensive 
approach  to  procuring  goods  and  services,  it  allows  the  government  to  pay  higher  prices 
for  reduced  perfonnance  risk,  and/or  greater  technical  capabilities,  which  may  provide 
better  overall  value.  According  to  the  FAR,  “a  tradeoff  process  is  appropriate  when  it 
may  be  in  the  best  interest  of  the  government  to  consider  award  to  other  than  the  lowest 

priced  offeror  or  other  than  the  highest  technically  rated  offeror”  (FAR  15.101-1). 
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The  tradeoff  method  of  source  selection  is  a  process  that  “allows  the  government 
the  flexibility  to  award  to  an  offeror  anywhere  on  the  best-value  continuum  between  the 
lowest  priced  technically  acceptable  offeror  and  the  highest  technically  rated  offeror” 
(Rendon  &  Snider,  2008,  p.  175).  The  FAR  gives  acquisition  professionals  a  broad 
discretion  of  which  evaluation  factors  and  significant  subfactors  can  be  used  in 
solicitations  so  long  as  these  factors  “are  tailored  to  the  acquisition,”  “represent  the  key 
areas  of  importance  and  emphasis  to  be  considered  in  the  source  selection  decision,”  and 
“support  meaningful  comparison  and  discrimination  between  and  among  competing 
proposals”  (FAR  15.304).  Non-cost  factors  that  are  typically  used  in  the  tradeoff  process 
include,  but  are  not  limited  to,  adherence  to  the  solicitation,  superior  technical  approach, 
relevant  and  recent  past  perfonnance,  small  business  subcontracting  plan,  professional 
qualification,  risk  reduction  plan,  and  the  ability  to  meet  or  exceed  the  schedule. 

D.  DEPARTMENT  OF  DEFENSE  SOURCE  SELECTION  PROCEDURES 

Regardless  of  the  source  selection  method  (LPTA  or  tradeoff)  used  to  acquire 
goods  and  services,  all  competitive  awards  under  FAR  15  must  follow  the  DOD 
standardized  Source  Selection  Procedures  (DOD,  2011).  In  November  2008,  John  J. 
Young,  the  Undersecretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
(USD[AT&L]),  established  the  Source  Selection  Joint  Analysis  Team  (JAT; 
USD[AT&L],  2011).  The  primary  charge  for  the  JAT  was  to  “standardize  the 
methodology  and  approach  the  Department  uses  to  conduct  competitively  negotiated 
source  selections”  (USD[AT&L],  201 1,  p.  1).  It  took  the  JAT  over  two  years  to  develop  a 
robust,  standardized  approach  to  conducting  source  selections;  the  use  of  these 
procedures  became  mandatory  across  the  DOD  as  of  July  2011  (USD[AT&L],  2011). 

The  Source  Selection  Procedures  give  acquisition  teams  a  start-to-finish  road  map 
of  the  entire  source  selection  process.  It  begins  with  a  statement  of  its  purpose,  outlines 
the  procedures,  defines  the  roles  and  responsibilities  of  each  source  selection  team 
member  and  provides  a  standardized  listing  of  all  documents  that  should  be  used  during 
the  process  (DOD,  2011).  When  referenced  properly,  the  DOD  Source  Selection 
Procedures  should  help  agencies  achieve  successful  source  selection  results,  shield  the 
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agencies  from  sustainable  protests,  and  help  achieve  the  goal  “to  ensure  the  Department’s 
source  selection  process  delivers  quality,  timely  products  and  services  to  the  Warfighter 
and  the  Nation  at  the  best  value  for  the  taxpayer”  (DOD,  201 1,  p.  1). 

E.  ACQUISITION  REFORM 

Prior  to  the  implementation  of  a  standardized  DOD  source  selection  guide,  the 
process  for  source  selection  and  the  concept  of  best  value  have  gone  through  several 
iterations  of  acquisition  reform  throughout  the  years.  Some  of  the  earlier  attempts  started 
in  1981  through  the  Carlucci  acquisition  refonn  initiatives  and  later  through  the  FAS  A, 
the  FARA,  and  the  Better  Buying  Power  (BBP)  initiatives  (Fox,  2011). 

One  of  the  early  attempts  to  redefine  and  clarify  the  best  value  concept  was 
through  the  FASA.  As  mentioned  earlier,  this  act  focuses  on  simplifying  and  refonning 
acquisition  laws  in  order  to  achieve  efficiencies  in  the  acquisition  process.  A  portion  of 
this  act  emphasized  the  concept  of  best  value,  which  allowed  agencies  to  award  contracts 
to  higher-priced  offerors  based  on  a  tradeoff  of  evaluation  factors  (FAR  15.101-1).  The 
act,  however,  missed  a  couple  of  important  issues  according  to  Fox  (2011),  “for  example, 
instead  of  giving  procuring  offices  the  freedom  to  use  innovative  methods  and  their  own 
judgment  to  obtain  the  ‘best  value’  for  their  organizations,  the  bill  included  new 
restrictions  and  oversight”  (p.  166).  Furthermore,  “it  provided  no  simplification  or 
streamlining  at  all  for  contracts  valued  over  $100,000”  (p.  166). 

While  the  FASA  was  a  good  starting  point  for  much  needed  acquisition  reforms, 
there  has  been  conflicting  emphasis  on  the  latitude  given  to  contracting  officers  in  source 
selections.  With  the  passage  of  the  FARA,  which  mandates  “that  full  and  open 
competition  in  government  contracting  be  implemented  in  a  way  that  is  consistent  with 
the  need  to  fulfill  efficiently  the  government’s  requirements”  (FARA,  n.d.,  para.  1), 
contracting  officers  gained  more  discretion  to  “limit  the  number  of  proposals  in  the 
competitive  range”,  which  promotes  source  selection  efficiency.  This  latitude  to  limit  the 
number  of  proposals  to  the  highest  rated  bidders  is  deemed  “one  of  the  most  significant 
changes  in  the  government  contracting  area  and  one  of  the  most  controversial”  (para.  2). 
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A  renewed  focus  on  source  selections  took  place  with  efforts  to  standardize  the 
process  across  the  DOD.  The  idea  to  create  the  JAT  was  prompted  by  a  report  to 
Congress  by  the  secretary  of  defense  on  the  Acquisition  Transformation  Initiatives  (OSD, 
2008).  One  of  the  recommendations  was  to  “change  existing  source  selection  guidance  to 
enhance  communication  to  industry”  in  order  to  “improve  everyone’s  knowledge  and 
understanding  of  the  process  and  seeks  to  avoid  protests  and  misunderstandings”  (p.  21). 
Prior  to  the  implementation  of  the  DOD  Source  Selection  Procedures,  each  service  used 
agency-specific  regulations  and  their  own  interpretations  of  FAR  15  procedures.  This 
practice  made  it  cumbersome  and  confusing  for  industry  wishing  to  participate  in  source 
selections  across  the  DOD.  For  example,  a  supplier  participating  in  multiple  source 
selections  across  different  agencies  and  navigating  the  apparent  and  not  so  apparent 
differences  in  the  way  each  agency  conducted  source  selections  could  have  been 
frustrated.  The  standardization  of  source  selection  procedures  across  the  DOD  is  also 
important  for  obtaining  better  research  data.  With  all  agencies  following  the  same 
procedures,  the  outcomes  of  each  source  selection  can  be  documented  and  studied  for 
potential  trends. 

The  Better  Buying  Power  (BBP)  initiatives  beginning  with  1.0  introduced  in  2010 
began  a  new  series  of  acquisition  reforms  focused  on  achieving  efficiency  and  improving 
acquisition  processes  across  the  DOD  (Carter,  2010).  It  was  not  until  BBP  2.0  launched 
in  2012,  however,  that  refonn  initiatives  addressed  best  value  concepts.  In  his 
memorandum,  Kendall  (2012)  outlines  how  this  can  be  achieved: 

The  Department  needs  to  improve  its  ability  to  define  the  value  to  the 
Department  of  performance  that  is  above  minimum  levels  so  that  it  can 
make  appropriate  source  selections  and  so  that  industry  can  bid 
intelligently.  This  will  spur  innovation  by  providing  a  predictable  basis  by 
which  companies  can  bid  enhanced  performance  with  the  knowledge  that 
any  increased  costs  are  within  an  acceptable  range,  (pp.  3-4) 

Recently,  the  DOD’s  focus  on  “best  value”  was  highlighted  again  in  the  release  of 
BBP  3.0.  This  update  sought  to  “provide  clear  and  objective  ‘best  value’  definitions  to 
industry”  (Kendall,  2015b,  p.  18).  This  effort  is  included  within  the  “Incentivize 
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Innovation  in  Industry  and  Government”  initiative,  and  Kendall  (2015b)  provides  the 
following  general  guidance: 

This  BBP  3.0  initiative  builds  on  the  work  started  in  BBP  2.0  to  provide 
industry  with  information  on  the  value,  in  monetary  terms,  of  higher  levels 
of  performance  than  minimally  acceptable  or  threshold  levels.  Without 
this  information,  the  default  position  will  be  to  bid  to  the  lowest 
acceptable  level  of  performance.  With  this  information,  industry  will 
know  what  the  competitive  effect  of  offering  higher  performance  will  be 
and  can  bid  accordingly,  (p.  18) 

Kendall  espoused  that  the  DOD  would  “focus  on  how  to  more  effectively 
monetize  best  value  and  publicize  relevant  case  studies”  (Kendall,  2015b,  p.18).  Specific 
actions  include:  (1)  updating  the  DOD  Source  Selection  Procedures  by  May  2015  to 
include  the  Best  Value  process  manual;  and  (2)  ensure  that  “best  value”  definitions  are 
“objective  and  stated  in  monetary  terms  as  much  as  possible”  (Kendall,  2015b,  p.18). 

In  addition  to  best  value  concepts,  the  use  of  LPTA  also  entered  the  spotlight 
based  on  concerns  from  industry.  Even  before  the  use  of  LPTA  was  discussed  in  BBP 
2.0,  the  drumbeat  from  LPTA  detractors  was  growing  steadily  louder.  To  address  these 
growing  concerns,  Kendall  followed  up  with  a  memorandum  titled  “Appropriate  Use  of 
Lowest  Priced  Technically  Acceptable  Source  Selection  Process  and  Associated  Contract 
Type”  (Kendall,  2015a).  The  bottom  line  of  his  memorandum  is  to  address  the  fact  that 
inappropriate  use  of  particular  source  selection  methods  can  undermine  the  government’s 
efforts  to  achieve  best  value  (2015a). 

F.  CONTROVERSY  OVER  USAGE 

Words  such  as  “sequestration,”  “economic  recession,”  and  “budget  deficit”  have 
been  main  topics  of  headline  news  for  several  years.  Young  (2015)  asserted  that  the  DOD 
budget  decreased  steadily  over  the  past  several  years  since  2010  due  to  the  winding  down 
of  conflicts  in  Iraq  and  Afghanistan  and  Congress  passing  the  Budget  Control  Act  of 
2011,  which  imposes  caps  on  discretionary  spending  also  known  as  sequester.  According 
to  Senator  Jeff  Sessions  (2011)  in  a  United  States  Senate  Budget  Committee  report,  the 
Budget  Control  Act  is  designed  to  cut  defense  spending  by  approximately  20%  while 
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increasing  non-defense  spending  by  about  50%,  “sequester  affects  defense  more  than 
other  programs”  (p.  1). 

In  response  to  the  national  budget  crisis,  the  fonner  USD(AT&L),  Ashton  Carter 
(2010),  emphasized  the  need  to  “do  more  without  more”  (p.  1)  in  Better  Buying  Power: 
Guidance  for  Obtaining  Greater  Efficiency  and  Productivity  in  Defense  Spending,  a 
memorandum  for  all  acquisition  professionals.  With  the  budget  getting  tighter  each  year, 
the  DOD  must  find  ways  to  achieve  efficiencies  across  the  board,  especially  within  the 
realm  of  contracting  processes  and  procedures.  While  it  is  unclear  whether  the  budget 
constraints  caused  the  increase  in  the  DOD’s  utilization  of  the  LPTA  source  selection 
method,  a  2014  GAO  report  indicated  that  “DOD  increased  its  use  of  LPTA  from  fiscal 
year  2009  to  fiscal  year  2013  for  contracts  with  obligations  of  $25  million  or  more” 
(p.  11).  The  DOD  is  also  reported  to  be  the  top  user  of  the  LPTA  procurement  method, 
according  to  Langan  (2013). 

This  increase  in  the  use  of  LPTA  by  government  procurement  offices  is  stirring 
up  controversy  between  industry  and  government.  As  cited  in  Gansler  and  Lucyshyn 
(2013),  “The  National  Defense  Industrial  Association  (2012)  named  improper  use  of 
LPTA  sourcing  as  one  of  its  ‘top  issues’  of  2012,  while  the  Professional  Services  Council 
(PSC)  called  monitoring  the  frequency  and  misuse  of  LPTA  a  ‘2013  Policy  Priority’” 
(p.  v).  According  to  the  GAO  (2014),  the  DOD  is  the  biggest  spender  of  the  federal 
budget  out  of  all  other  departments;  it  “obligated  about  $310  billion  in  fiscal  year  2013  to 
acquire  products  and  services  needed  to  support  its  missions”  (p.  1).  This  fact  puts  the 
DOD’s  acquisition  practices  under  a  microscope,  leading  to  high  levels  of  scrutiny  and 
criticism  from  all  stakeholders,  especially  the  defense  industry  and  Congress.  The  issues 
surrounding  the  use  of  LPTA  are  making  greater  noise  in  the  media  and  becoming  the 
subject  of  leadership  attention  in  recent  years. 

1.  Is  LPTA  Used  Appropriately? 

With  increased  use  of  the  LPTA  method  for  contracts  above  $25  million,  critics 
(mainly  defense  contractors)  question  whether  such  methods  are  being  used 
appropriately.  FAR  15.101  outlines  the  various  methods  within  the  Best  Value 
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Continuum  whereby  agencies  can  choose  “any  one  or  a  combination  of  source  selection 
approaches”  (FAR  15.101)  to  obtain  best  value.  While  the  FAR  outlines  that  “in  different 
types  of  acquisitions,  the  relative  importance  of  cost  or  price  may  vary”  (15.101),  the 
ultimate  decision  to  use  either  the  LPTA  or  tradeoff  method  depends  on  the  contracting 
officer’s  discretion.  To  provide  further  guidance  on  the  issue,  Kendall  (2015a)  pointed 
out  in  his  DOD-wide  memorandum  that 

LPTA  is  the  appropriate  source  selection  process  to  apply  only  when  there 
are  well-defined  requirements,  the  risk  of  unsuccessful  contract 
performance  is  minimal,  price  is  a  significant  factor  in  the  source 
selection,  and  there  is  neither  value,  need,  nor  willingness  to  pay  for 
higher  performance,  (p.  1) 

Additionally,  the  new  DOD  (2011)  Source  Selection  Procedures  suggest  that 

LPTAs  may  be  used  in  situations  where  the  govermnent  would  not  realize 
any  value  from  a  proposal  exceeding  the  government’s  minimum  technical 
or  performance  requirements,  often  for  acquisitions  of  commercial  or  non¬ 
complex  services  or  supplies  which  are  clearly  defined  and  expected  to  be 
low  risk.  (p.  A-l) 

Although  there  are  published  policies  and  guidance  regarding  the  appropriate  use  of 
LPTA,  Weckstein  and  Delgado  (2012)  claimed  that  “the  legal  framework  on  use  of 
LPTA  remains  unworkably  vague”  (p.  2)  and  that  there  is  no  “firm  standard  for  when 
LPTA  can  and  cannot  be  used”  (p.  3).  For  instance,  it  is  up  to  the  contracting  officer’s 
discretion  to  decide  the  source  selection  method  based  on  how  complex  and  or  risky  he  or 
she  perceives  a  project  to  be.  There  is  a  lack  of  clear  guidance  about  what  makes  a 
contract  “complex”  or  “risky.”  Weckstein  and  Delgado  (2012)  also  argued  that  “there 
must  be  some  limit  to  a  contracting  officer’s  discretion,  a  point  at  which  the  decision  to 
use  LPTA  sourcing  becomes  unreasonable  in  relation  to  agency  needs  and  the 
government’s  interest”  (p.  3). 

Aside  from  a  perceived  lack  of  clarity  in  prescribing  the  proper  use  of  various 
source  selection  methods,  the  DOD  is  criticized  for  using  LPTA  “where  marginal  product 
quality  is  a  differentiator  (such  as  medical  products  used  to  care  for  soldiers)  and  in  the 
outcome-based  services  industry”  (Crusius,  2015,  para.  3).  Referring  to  the  government’s 
use  of  LPTA  to  procure  such  services,  Crusius  (2015)  made  a  comparison  by  asking  the 
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question  “Would  you  like  it  if  your  children’s  school  hired  its  teachers  on  an  LPTA 
basis?”  (para.  3).  Gansler  and  Lucyshyn  (2013)  also  voiced  their  concern  about  the  fact 
that  “LPTA  source  selection  has  been  expanded  to  the  procurement  of  complex  hardware 
and  high-knowledge-content  professional  services”  (p.  9).  They  stated  that  the  use  of 
LPTA  “can  be  inappropriate  at  times,  especially  when  the  technology  in  question  is 
complex  or  unprecedented,  or  when  lives  are  at  stake”  (p.  4).  Gansler  and  Lucyshyn 
(2013)  also  argued  that  high-end,  complex  services  deserve  a  “thorough  examination  of 
‘trade-offs’  between  cost  and  non-cost  factors”  (p.  2).  Finally,  Lohfeld  (2012)  remarked 
that  “procurements  get  into  trouble  when  the  LPTA  criteria  is  applied  to  technical  and 
professional  services  bids  because  the  work  is  complex,  minimum  acceptable  technical 
and  performance  requirements  are  difficult  to  describe,  and  the  consequences  from  failure 
can  be  considerable”  (p.  1). 

Many  of  the  articles,  including  Gansler  and  Lucyshyn’s  (2013)  study,  point  to  the 
State  Department’s  personal  security  contract  for  the  U.S.  Embassy  in  Kabul, 
Afghanistan,  as  an  example  of  a  complex  service  procured  through  the  LPTA  method, 
which  should  have  been  acquired  through  tradeoff.  According  to  the  Commission  on 
Wartime  Contracting  in  Iraq  and  Afghanistan  (CWC,  2009), 

unlike  other  federal  agencies,  the  U.S.  Department  of  State  is  forbidden  by 
law  to  select  anything  but  the  lowest  price  and  ‘technically  acceptable’ 
offer  when  awarding  contracts  to  protect  its  overseas  buildings — even  if 
this  means  passing  up  offers  from  firms  offering  higher  quality  and  better 
experience.  In  contingency  operations  like  those  in  Iraq  and  Afghanistan, 
this  prohibition  can  have  negative  consequences  for  security,  wartime 
mission  objectives,  and  America’s  image,  (p.  1) 

Accordingly,  the  CWC  (2009)  reported  that  the  winning  bid  was  $110  million 
lower  than  its  competitor  and  the  contract  was  plagued  with  a  “litany  of  complaints  about 
poor  contract  performance  and  behavior  by  AGNA,  the  company’s  in-country  leadership 
team  was  slow  to  respond  to  problems  and  improperly  performed  their  duties  as  a 
contingency  contractor”  (p.  4).  The  contractor’s  performance  deficiencies  consisted  of 
failing  to  provide  the  proper  number  of  trained,  qualified  guards;  inability  of  the  guards 
to  communicate  with  embassy  personnel  and  visitors  due  to  language  barriers;  and 
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unacceptable  food  services,  inadequate  documents,  training,  and  unifonns  for  new 
employees,  which  were  all  requirements  of  the  contract  (CWC,  2009) 

High  profile  cases  such  as  the  State  Department’s  Kabul  embassy  example  along 
with  a  few  others  such  as  the  Air  Force’s  KC-X  program  and  the  Navy’s  next  generation 
enterprise  network  contract  discussed  by  Gansler  and  Lucyshyn  (2013),  leave  opponents 
of  LPTA  with  a  perception  that  the  method  is  widely  and  inappropriately  used  across  the 
DOD.  However,  the  2014  GAO  report  titled  Defense  Contracting:  Factors  DOD 
Considers  When  Choosing  Best  Value  Processes  Are  Consistent  with  Guidance  for 
Selected  Acquisitions  paints  a  much  different  picture.  According  to  the  results  of  this 
study,  the  “DOD  used  the  tradeoff  process  most  often  in  our  sample  to  acquire  services, 
including  those  related  to  construction  projects,  aircraft  maintenance,  and  other  support 
services,  regardless  of  obligation  amount”  (GAO,  2014,  p.  10).  The  report  also  claims 
that  the  LPTA  method  was  used  primarily  for  the  purchase  of  “commercial  products  such 
as  fuel”  (p.  10)  for  contracts  at  or  above  $25  million.  While  it  is  true  that  the  use  of  LPTA 
in  DOD  contracts  increased  over  the  past  few  years,  its  application  is  found  to  be 
consistent  with  guiding  policies  and  procedures  appropriate  for  the  level  of  risk  and 
complexity  of  the  projects  awarded  under  its  method. 

2.  Does  LPTA  Provide  Best  Value? 

The  biggest  issue  with  increased  use  of  the  LPTA  method  is  that  it  is  perceived  by 
critics  to  be  the  cheapest,  short-term  focused  solution,  which  is  not  a  best  value  approach 
to  procurement.  According  to  Meagher  (2012),  “Lowest  Price  Technically  Acceptable  is 
commonly  viewed  as  an  alternative  to  best  value”  (para.  2)  even  though  the  FAR  clearly 
outlines  that  it  is  one  of  the  methods  that  can  be  used  to  achieve  best  value.  The  FAR 
specifically  states  that  LPTA  is  appropriate  “when  best  value  is  expected  to  result  from 
the  selection  of  the  technically  acceptable  proposal  with  the  lowest  evaluated  price” 
(FAR  15.101-2).  Yet  industry  argues  that  best  value  cannot  be  achieved  when  price 
becomes  the  biggest  discriminator,  with  technical  capabilities  and  company  experience 
seated  in  the  back  on  a  pass  or  fail  basis. 
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With  regard  to  best  value,  Gansler  and  Lucyshyn  (2013)  asserted  that  ignoring 
such  important  non-cost  factors  and  focusing  on  lowest  price  could  lead  to  awarding 
contracts  to  Finns  with  “fewer,  or  less  experienced,  employees;  unreliable  supply  chains; 
or  less  effective  quality  control  processes — factors  that  may  not  be  evident  in  the 
proposal,  and,  even  if  they  are,  such  factors  are  typically  excluded  from  consideration 
under  LPTA”  (p.  2).  According  to  Lohfeld  (2015),  “overemphasizing  cost  causes  bidders 
to  be  much  more  aggressive  at  risk  taking,  and  bidders  will  understate  their  cost  of 
performance  in  an  effort  to  be  the  lowest  priced,  technically  acceptable  bidder”  (para.  4). 
Lohfeld  (2015)  also  emphasized  that  “in  the  long  run,  expected  cost  savings  are  erased  by 
cost  overruns,  contract  modifications,  show  cause  letters,  and  even  contract  terminations” 
(para.  4).  When  contractors  are  forced  to  compete  on  the  basis  of  lowest  price,  Perkins 
(2014)  opined,  this  drives  contractors  to  hire  the  cheapest  people  they  can  possibly  find 
and  that  “projects  staffed  via  LPTA  usually  end  up  with  the  least  expensive  people 
meeting  minimum  levels  of  education,  security  clearance  and  so  on.  Such  people  are 
likely  to  have  trouble  with  unexpected  or  complex  problems”  (para.  4).  Most  companies 
in  the  defense  industry  seem  to  agree  that  “the  level  of  quality  or  the  value  provided  to 
the  government  suffers”  (Anderson,  2014,  para.  2)  when  projects  are  awarded  under 
LPTA. 

Industry  often  argues  that  not  only  does  LPTA  rob  the  government  of  long-term 
value  by  ignoring  important  non-cost  factors,  but  that  the  level  of  customer  service, 
overall  effort,  and  quality  in  the  short-run  is  also  subpar.  With  all  the  profits  “squeezed” 
out  of  the  contractors,  anything  above  the  minimum  requirement  becomes  unaffordable 
(Calisti,  2015).  Anderson  (2014)  compared  services  received  under  LPTA  to  installing 
the  cheapest  tires  for  a  car,  “they  may  not  last  very  long,  and  they  could  have  blowouts 
that  are  not  covered  by  warranties,  they  could  cause  vibrations  at  highway  speeds  due  to 
imbalance  but  they  will  allow  the  car  to  roll  down  the  road”  (para.  2).  Anderson  (2014) 
also  compared  LPTA  to  achieving  a  “D”  in  school,  which  is  the  lowest  passing  grade, 
and  encouraged  govermnent  contractors  to  “be  diligent  in  reviewing  the  solicitation  to  see 
if  the  evaluation  will  be  LPTA  or  Best  Value”  (para.  4);  if  it  is  LPTA,  find  out  exactly 
what  the  “bare  minimum”  technical  acceptable  level  is  and  strive  for  a  D  (Anderson, 
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2014).  With  such  a  stance,  the  defense  industry  believes  that  quality,  creativity,  and 
innovation  have  no  place  in  a  contract  awarded  under  the  LPTA  method. 


3.  LPTA  Leaves  No  Room  for  Innovation 

Another  consensus  among  industry  with  regard  to  increased  use  of  LPTA  is  that  it 
leaves  no  incentive  for  companies  to  innovate.  With  smaller  profit  margins,  companies 
cannot  afford  to  bring  creative  and  innovative  solutions  to  the  table.  In  a  Washington 
Technology  article,  Lisa  Mascolo  (2012)  referred  to  LPTA  as  “the  antithesis  of 
innovative”  (para.  3).  Innovation  plays  a  crucial  role  for  both  the  DOD  as  well  as  the 
nation.  During  an  interview,  Zachary  Fryer-Biggs  (2012)  captured  a  quote  from  the 
former  U.S.  defense  secretary,  Leon  Panetta,  on  the  importance  of  innovation: 

We’re  depending  a  great  deal  on  being  at  the  technological  edge  of  the 
future,  Panetta  said.  I  think  we  even  have  to  leap  forward.  If  we’re  going 
to  deal  with  the  kind  of  challenges  we’re  going  to  face,  we’ve  got  to  be 
smart  enough,  innovative  enough,  creative  enough  to  be  able  to  leap 
forward,  (para.  6) 

Innovation  is  also  the  heart  of  a  nation’s  economic  development  and  evolution.  In 
a  world  of  global  markets,  innovation  is  also  the  key  to  obtaining  competitive  advantage. 
According  to  Gansler  and  Lucyshyn  (2013),  “it  is  clear  that  global  markets;  technology 
proliferation;  and  education  in  science,  technology,  engineering  and  mathematics  will 
drive  innovation  in  defense  and  other  increasingly  global  industries”  (pp.  21-22). 
However,  Gansler  and  Lucyshyn  (2013)  worried  that  “increased  reliance  on  LPTA  may 
also  accelerate  worrisome  national  trends,”  (p.  21)  where  “American  dominance  in 
scientific  and  technological  innovation  will  continue  to  decline”  (p.  22).  Gansler  and 
Lucyshyn  (2013)  further  asserted  that  “reducing  the  demand  for  new  innovation  could 
potentially  weaken  America’s  economy  and  national  security  posture”  (p.  22). 

4.  Impact  of  LPTA  on  the  Defense  Industry 

While  defense  spending  has  been  cut  across  the  board,  the  number  of  companies 
in  the  industry  looking  to  win  bids  has  not  changed  (Weckstein  &  Delgado,  2012).  With 
less  money  to  spread  around,  competition  within  the  industry  has  been  heating  up  over 

the  past  few  years  (Burnett,  2011).  Additionally,  the  use  of  LPTA  source  selection 
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methods  only  seem  to  fuel  the  fire,  further  straining  companies  in  the  industry,  forcing 
them  to  take  drastic  measures  such  as  reducing  employee  salaries  and  layoffs  (Perkins, 
2014).  For  instance,  according  to  an  article  by  Censer  (2014),  Wayne  Luceroni,  the 
president  of  Harris  IT  services,  issued  a  statement  during  an  interview  that  his  company 
has  cut  costs,  including  consolidating  seven  Northern  Virginia  offices  into  one  facility  in 
order  to  “cope  with  what  industry  views  as  increasing  price  pressure”  (para.  16). 
Furthermore,  Young  (2015)  claimed  that 

in  this  new  LPTA  business  environment,  savvy  management  teams  have 
been  reorganizing  their  business  development  teams  and  delivery 
capabilities  around  new  standards.  Operationally,  contractors  have 
stripped  out  all  sorts  of  costs;  reducing  their  wrap  rates  and  overall  cost  of 
service  delivery  and  product  production.  In  terms  of  business 
development,  many  contractors  have  turned  to  outside  consultants  to  assist 
with  ad-hoc  proposal  development.  Their  efforts  have  yielded  significantly 
lower  Selling  General  and  Administrative  (SG&A)  rates,  which  benefited 
the  government  financially  on  both  fixed  price  and  cost-plus  programs. 

(P- 1) 

Young  (2015)  also  asserts  that  the  outsourcing  of  proposal  development  by 
contractors  is  creating  new  problems  for  both  government  and  contractor,  where 
contractors  are  winning  bids  but  unable  to  deliver  perfonnance  because  they  did  not  write 
the  proposals.  As  a  response,  the  Department  of  Veterans  Affairs  (VA)  banned  proposal 
preparation  by  outside  consultants  on  November  19,  2014,  which  was  protested  by  a 
small  business  on  grounds  that  the  ban  was  “unduly  restrictive”  (p.  1).  While  the  protest 
was  denied  by  the  GAO,  “the  implications  for  small  and  middle-market  contractors  are 
serious.  If  a  company  cannot  afford  to  maintain  full-time  business  development 
employees,  it  will  be  at  a  severe,  if  not  insurmountable,  disadvantage  for  competing  on 
RFPs”  (Young,  2015,  p.  2).  While  the  VA’s  concerns  over  contract  perfonnance  risk  are 
legitimate,  the  GAO’s  decision  would  force  smaller  finns  with  limited  resources  into 
teaming  agreements  while  larger  primes  benefit  (Young,  2015). 

5.  Leadership  Awareness  of  LPTA  Issues 

Proponents  of  LPTA  believe  that  all  the  issues  discussed  thus  far  are  just  tactics 
used  by  industry  to  “demonize”  the  source  selection  method  because  it  further  nanows  a 
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profit  margin  already  impacted  by  budget  cuts  (Cox,  2015).  Cox  (2015)  claimed  that 
“contractors  would  prefer  that  DOD  use  a  more  expensive  process  called  ‘best  value,’ 
which  allows  them  to  charge  more  for  features  that  warfighters  usually  don’t  need,” 
(para.  4)  and  further  insists  that  all  the  industry  complaints  published  in  trade  reports  are 
“unfounded”  (para.  9).  The  biggest  issue  over  the  use  of  LPTA  among  industry  is  that  it 
leads  to  low  quality  performance.  According  to  Lohfeld  (2012),  the  key  to  avoiding 
quality  issues  with  the  LPTA  method  is  to  ensure  that  the  standards  to  meet  the  minimum 
technical  acceptability  is  set  at  a  level  that  meets  the  customer’s  needs.  Furthermore,  cost 
and  price  analysis  techniques  such  as  price  realism  (price  is  too  low)  and  price 
competitiveness  (price  is  too  high)  can  be  utilized  to  evaluate  proposals  that  seem  too  low 
or  risky  (Garrett,  2015).  The  controversy  over  the  use  of  LPTA,  however,  is  not  ignored 
by  Congress  or  the  leaders  in  the  DOD  acquisition  community.  According  to  the  House 
Armed  Services  Committee  (HASC)  Report  232-33  (as  cited  in  Nichols,  2013), 

the  House  Armed  Services  Committee  would  have  the  U.S.  Comptroller 
General  conduct  a  review  of  [DOD’s]  use  of  source  selection  processes, 
including  LPTA-and,  [i]n  conducting  the  review,  ...  to  obtain  the  views  of 
defense  contractors  to  gain  insight  into  how  the  use  of  LPTA  source 
selection  procedures  affects  business  decisions  and  to  identify  the 
unintended  consequences,  if  any,  resulting  from  the  use  of  this  approach. 

(P-  1) 

The  HASC  Report  further  “directs  the  Comptroller  General  to  provide  the 
findings  of  the  review,  along  with  recommendations  to  improve  the  Department’s 
contracting  practices,  to  the  congressional  defense  committees  by  June  30,  2014”  (as 
cited  in  Nichols,  2013,  p.  1).  In  addition  to  the  Congressional  inquiries,  Jared  Serbu 
(2014)  reported  that  the  USD(AT&L),  Frank  Kendall,  tasked  “Randall  Culpepper,  the  Air 
Force’s  top  official  for  services  contracting”  (para.  3)  to  lead  a  review  of  LPTA  usage 
across  the  DOD. 

G.  SUMMARY 

This  chapter  explored  general  information  on  statutes  and  regulations  that 
influence  federal  contracting  and  reviewed  the  phases  of  the  contract  management 
process.  Additionally,  the  difference  between  LPTA  and  tradeoff  source  selection 
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methods  were  examined  along  with  the  concepts  of  best  value  and  the  best  value 
continuum.  The  chapter  also  reviewed  the  new  standardized  DOD  Source  Selection 
Procedures,  as  well  as  some  history  on  previous  efforts  to  reform  the  federal  acquisition 
process  and  best  value  principles.  Finally,  some  controversial  issues  regarding  the 
increased  use  of  LPTA  were  discussed  to  set  the  stage  for  why  this  research  is  being 
conducted.  The  next  chapter  describes  the  methodology  for  this  study,  including 
background  information  regarding  the  Air  Force  Life  Cycle  Management  Center, 
methods  used  to  identify  the  data,  and  how  the  data  was  obtained. 
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III.  METHODOLOGY 


This  chapter  will  discuss  the  methodology  used  to  locate,  collect,  and  analyze  data 
in  order  to  answer  the  research  questions.  This  includes  a  description  of  the  data  source, 
the  types  of  data  accessed,  and  the  data  collection  process  which  describes  the  criteria 
used  for  selecting  contract  files  and  the  data  collection  rubric.  The  last  section  describes 
the  analysis  protocol  that  was  used  to  quantify  the  relationship  (if  any)  between  source 
selection  method  and  contract  outcomes  such  as  CPARS  ratings  and  PALT. 

A.  SOURCE  OF  DATA 

For  this  research,  we  selected  Wright-Patterson  AFB  as  the  source  of  data  because 
of  its  perceived  prominence  in  Air  Force  acquisition.  Wright-Patterson  AFB  is  home  to 
both  the  Air  Force  Materiel  Command  (AFMC)  and  Air  Force  Life  Cycle  Management 
Center  (AFLCMC)  which  are  involved  in  the  acquisition  and  sustainment  of  major 
weapons  systems  to  support  the  Air  Force  mission.  To  most  members  of  the  Air  Force 
acquisition  community,  Wright-Patterson  represents  the  epicenter  of  procurement  and 
acquisition  activity. 

On  June  4,  2010,  a  memorandum  was  issued  by  the  Secretary  of  Defense, 
challenging  services  to  “eliminate  redundant  functions  and  increase  funding  for  mission 
functions  through  efficiency  savings”  (AFLCMC,  2015,  P.  10).  The  Air  Force  believed 
the  most  prudent  approach  to  meeting  the  SECDEF’s  charge  was  a  restructuring  of  the 
command.  The  decision  was  made  that  a  reduction  from  12  location-specific 
organizations,  each  requiring  a  full  staff,  down  to  five  mission-based  life-cycle  centers 
would  be  the  appropriate  way  ahead.  The  “Five  Center  Construct”  as  it  was  dubbed,  was 
designed  to  “reduce  overhead  cost  and  eliminate  redundant  layers  of  staff’  (AFLCMC, 
2015,  p.  1 1).  It  was  also  believed  that  such  a  structure  would  allow  for  more  standardized 
business  practices  and  provide  customers  with  a  “single  face”  for  their  programs 
(AFLCMC,  2015,  p.  11).  AFLCMC  is  one  of  the  centers  that  fonned  as  a  result  of  the 
restructuring. 
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The  AFLCMC  was  designed  to  be  a  “mechanism  to  bring  focused,  disciplined 
and  integrated  life  cycle  management  to  the  acquisition  of  Air  Force  weapon  systems” 
(AFLCMC,  2015,  p.  11).  The  overarching  goal  was  to  achieve  savings  and  better 
performance  by  managing  the  entire  life  cycle  of  a  system  as  opposed  to  its  individual 
parts  such  as  development,  testing,  production,  and  support.  The  belief  was  that  a  single 
center  would  provide  unifonn  and  continuous  process  improvement  at  an  enterprise  level. 
The  first  AFLCMC  commander,  Lieutenant  General  Clyde  D.  Moore,  developed  the 
following  mission  statement  for  the  new  center:  “Delivering  affordable  and  sustainable 
war-winning  capabilities  to  the  U.S.  and  international  partners  on  time,  on  cost, 
anywhere,  anytime  from  cradle  to  grave” 

For  this  research,  contracts  from  three  of  the  10  AFLCMC  directorates  were 
reviewed.  The  three  directorates  were  the  Agile  Combat  Support  (ACS)  Directorate,  the 
Mobility  Directorate,  and  the  Contract  Execution  Directorate.  ACS  oversees  Air  Force 
programs  to  include  metrology  and  calibration,  automatic  test  systems,  combat 
electronics,  electronic  warfare,  and  avionics  and  simulators  (AFLCMC,  2015,  p.  15).  The 
ACS  contracts  reviewed  for  this  study  were  all  within  the  simulator  section.  The  Mobility 
Directorate  manages  the  “development,  production,  testing,  deployment,  and  sustainment 
of  a  mobility  and  training  aircraft”  (AFLCMC,  2015,  p.  17).  Finally,  the  Contract 
Execution  Directorate  (PZI)  office  serves  as  an  organization  that  provides  contractual 
support  to  the  various  directorates  spread  across  the  AFLCMC  and  is  primarily  focused 
on  providing  infonnation  technology  solutions. 

B.  TYPES  OF  DATA 

Numerous  electronic  federal  acquisition  systems  exist,  in  part,  due  to  federal 
statutes,  regulations,  policies,  and  in  response  to  process  improvement  initiatives.  These 
systems  serve  several  functions  including  data  collection,  data  dissemination, 
documenting  contractor  performance,  and  contract  document  execution.  The  systems 
accessed  in  support  of  this  research  include  the  Contract  Writing  System  (ConWrite),  the 
Federal  Procurement  Data  System-Next  Generation  (FPDS-NG),  the  Contractor 
Performance  Assessment  Reporting  System  (CPARS),  and  the  Earned  Value  Central 
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Repository.  This  section  explains  each  system  in  further  detail  and  highlights  the  reasons 
why  they  were  accessed  for  this  study. 

1.  Contract  Writing  System 

ConWrite  is  one  of  several  contract  writing  systems  used  by  the  United  States  Air 
Force  (USAF).  ConWrite  is  a  contract  document  preparation  software  package  that 
allows  users  to  prepare  contracts,  solicitations,  grants,  modifications,  and  orders  for  Air 
Force  Materiel  Command  (AFMC),  Air  Force  Space  Command  (AFSPC)  Product  and 
Test  Centers,  and  the  Air  Force  Research  Laboratory  (B&ESD,  2015,  p.  68).  For  this 
research,  ConWrite  was  used  to  generate  a  report  identifying  contracts  that  met  certain 
parameters.  The  report  was  then  disseminated  to  several  AFLCMC  directorates  to  verify 
whether  the  contract  files  were  available  for  review. 

2.  Federal  Procurement  Data  System-Next  Generation 

FPDS-NG  is  “a  computer-based  Federal  Procurement  Data  System  for  collecting, 
developing  and  disseminating  procurement  data  to  the  Congress,  Executive  Branch,  and 
private  sector”  (FPDS,  2015,  p.  1).  Information  is  entered  into  the  system  through  a 
contract  action  report  (CAR)  once  a  contract  action  is  completed.  The  system  contains 
comprehensive  post-award  information  pertaining  to  individual  contracts  awarded  within 
the  federal  government.  This  information  includes  inputs  such  as  contract  type,  the  extent 
to  which  the  contract  was  competed,  and  the  number  of  offerors,  among  others.  For  the 
purposes  of  this  research,  FPDS-NG  was  used  to  generate  a  list  of  contracts  that  could  be 
reviewed  during  the  data  collection  process. 

3.  Contractor  Performance  Assessment  Reporting  System 

CPARS  is  used  to  document  contractor  performance  and  “is  one  of  the  tools  used 
to  communicate  contractor  strengths  and  weaknesses  to  source  selection  officials  and 
Contracting  Officers”  (CPARS,  2015,  p.  v).  Contractor  performance  is  assessed  through 
five  broad  evaluation  areas  including  Quality,  Schedule,  Cost  Control,  Management,  and 
Utilization  of  Small  Business,  if  applicable  to  the  contract  (CPARS,  2015). 
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CPARS  reporting  requirements  differ  between  civilian  federal  agencies  and  the 
DOD.  Although  federal  civilian  agencies  require  CPARS  reports  for  systems  and  non¬ 
systems  contracts  above  the  simplified  acquisition  threshold  (SAT),  the  DOD  typically 
requires  CPARS  reports  for  contracts  at  higher  dollar  thresholds  (CPARS,  2015). 
Appendix  A  outlines  the  DOD  CPARS  requirements  based  on  business  sector  and  dollar 
threshold. 

The  DOD  CPARS  requirements  were  useful  to  detennine  search  parameters  for 
contracts  in  ConWrite  and  FPDS-NG.  CPARS  was  accessed  for  this  research  because 
contractor  performance  data  is  an  important  post- award  outcome. 

4.  Earned  Value  Central  Repository  System 

Earned  Value  Management  (EVM)  is  a  program  management  practice  to  assess 
program  status  through  cost,  schedule,  and  technical  performance  (PARCA,  2015). 
Further,  the  Earned  Value  Management  System  (EVMS)  “is  an  integrated  management 
system  that  integrates  the  work  scope,  schedule,  and  cost  parameters  of  a 
program  in  a  manner  that  provides  objective  performance  measurement  data” 
(OUSD  [  AT &L]/P  ARC  A,  2015). 

The  DOD  requirements  for  EVM  are  found  in  the  Defense  Federal  Acquisition 
Regulation  Supplement  (DFARS)  Subpart  234.2  and  the  DOD  Instruction  (DODI) 
5000.02  (OUSD[AT&L],  2015).  The  EVM  reporting  requirements  are  based  on  contract 
type  and  dollar  threshold  in  then-year  dollars  (Appendix  B).  Referencing  the  EVM 
requirements  was  useful  to  determine  search  parameters  for  contracts  in  ConWrite  and 
FPDS-NG.  The  Earned  Value  Central  Repository  was  accessed  for  this  research  because 
EVM  data  is  an  important  post-award  perfonnance  outcome.  It  is  indicative  of  a 
contractor’s  ability  to  manage  the  cost,  schedule,  and  technical  performance  aspects  of  a 
program  (PARCA,  2015). 

C.  DATA  COLLECTION 

The  data  collection  process  involved  travelling  to  Wright-Patterson  AFB  to 
physically  review  contract  files.  Contracts  were  identified  by  working  with  AFLCMC  to 
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run  a  query  in  both  ConWrite  and  FPDS-NG  based  on  particular  parameters.  This  section 
explains  the  efforts  made  to  identify  and  locate  contract  files  and  the  spreadsheet  rubric 
used  to  collect  the  necessary  data. 

1.  Criteria  for  Identifying  Contract  Files 

Reports  were  generated  using  ConWrite  and  FPDS-NG  to  identify  contracts  for 
review.  Initially,  AFLCMC  Contracting  Directorate  (AFLCMC/PK)  queried  ConWrite 
based  on  the  following  criteria:  (1)  Contracts  closed  within  the  past  calendar  year  (2015), 
(2)  contract  value  >$1  million,  and  (3)  ideal  sample  of  40-60  contracts.  AFLCMC/PK 
was  advised  that  if  the  initial  report  did  not  result  in  a  list  of  40-60  contracts  based  on  the 
aforementioned  criteria,  an  additional  query  could  be  accomplished  to  include  all 
contracts  that  were  within  the  last  year  of  contract  perfonnance  (calendar  year  2015).  The 
resulting  report  included  a  total  of  70  contracts  that  were  closed  within  the  past  two  years. 
This  report  was  then  forwarded  to  the  following  AFLCMC  directorates  to  identify  and 
locate  any  of  the  contract  files:  Business  &  Enterprise  Systems  (HI),  Contract  Execution 
(PZ),  Air  Force  Security  Assistance  Cooperation  (WF),  Agile  Combat  Support  (WN), 
645th  Aeronautical  Systems  Group  (WI),  Mobility  (WL),  and  Fighters  and  Bombers 
(WW).  Unfortunately,  many  of  the  contracts  listed  in  that  report  were  unavailable  for 
review  because  they  were  either  in  staging  or  they  were  noncompetitive  contracts. 
ConWrite  does  not  offer  the  ability  to  query  based  on  competitive  versus  non-competitive 
contracts.  To  expand  the  potential  amount  of  contracts  to  review,  AFLCMC/PK  provided 
an  updated  report  that  included  all  AFLCMC  contracts  that  were  awarded  in  calendar 
years  2010  and  2011.  This  report  was  much  more  extensive  and  included  910  contracts. 
The  report  was  filtered  and  sorted  for  contracts  valued  greater  than  $1  million,  which 
resulted  in  a  list  of  55  contracts.  This  report  was  also  forwarded  to  the  aforementioned 
directorates.  Since  reports  generated  from  ConWrite  could  not  identify  whether  contracts 
were  competitive,  a  third  report  was  generated  using  FPDS-NG.  AFLCMC/AZ  provided 
the  report  for  all  contracts  awarded  between  FY2010-2013,  which  was  then  filtered  by 
competitive  contracts,  resulting  in  a  list  of  188  contracts.  All  three  reports  were 
forwarded  to  the  directorates  to  identify  and  locate  contracts  to  review. 
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2. 


Data  Collection  Rubric 


The  data  collection  rubric  is  a  Microsoft  Excel  spreadsheet  created  to  capture  up 
to  98  inputs  per  observation  (contract  reviewed).  This  is  the  same  spreadsheet  used  in  the 
studies  conducted  by  Watson  (2015)  and  Bastola,  Findley,  and  Woodward  (2015).  The 
data  inputs  fell  within  five  broad  categories  related  to  the  pre-award  and  post-award 
contract  management  phases.  The  categories  are:  (1)  Basic  Information,  (2)  Acquisition 
Complexity,  (3)  Environmental  Factors,  (4)  Outcome  Variables,  and  (5)  Other.  This 
section  describes  each  broad  category  in  further  detail. 

The  Basic  Information  category  includes  descriptive  infonnation  for  each 
observation  such  as  the  agency,  the  contract  number,  the  name  of  the  program  or 
requirement,  and  the  NAICS  code.  This  category  includes  a  total  of  seven  data  inputs. 

The  Acquisition  Complexity  category  includes  variables  such  as  the  contract  type, 
dollar  value,  source  selection  method  (LPTA  or  tradeoff),  number  of  evaluation  factors, 
number  of  contract  line  item  numbers  (CLINs),  number  of  offerors,  and  whether  single  or 
multiple  award.  This  category  included  a  total  of  41  potential  data  inputs. 

The  Environmental  Factors  category  includes  variables  such  as  the  number  of 
reviews,  number  of  personnel  on  the  source  selection  team,  the  number  of  source 
selection  team  locations,  and  the  use  of  collaboration  tools  among  others.  This  category 
includes  a  total  of  13  potential  data  inputs. 

The  Outcome  Variables  category  included  the  procurement  administrative  lead 
time  (PALT)  infonnation  encompassing  the  requirement  receipt  date,  RFP  issue  date, 
proposals  received  date,  and  contract  award  date.  Additionally,  other  inputs  include 
whether  there  was  a  protest  (Yes  or  No),  the  level  of  the  protest  (if  applicable),  CPARS 
ratings,  and  EVM  infonnation.  This  category  includes  a  total  of  30  potential  data  inputs. 

The  Other  category  includes  variables  such  as  the  number  of  evaluation  notices, 
the  number  of  rounds  of  evaluation  notices,  number  of  deficiency  reports,  and  the  basis 
for  award.  This  category  includes  a  total  of  seven  potential  data  inputs. 
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D.  DATA  ANALYSIS 

This  research  aims  to  determine  whether  contract  performance  outcomes  such  as 
CPARS  ratings,  PALT,  and  EVM  (if  applicable)  are  impacted  by  the  choice  of  using  the 
LPTA  or  the  tradeoff  source  selection  method.  This  research  utilized  the  same  analysis 
technique  used  by  Watson  (2015)  and  Bastola  et  al.  (2015). 

For  this  analysis,  we  used  two  main  dependent,  or  outcome,  variables  (DVs)1: 
procurement  administrative  lead-time  (PALT)  and  Contractor  Performance  Assessment 
Rating  System  (CPARS)  data.  PALT  assesses  “time  to  contract”  by  calculating  the 
number  of  days  between  receipt  of  the  requisition  and  contract  award.  It  is  a  continuous 
variable.  CPARS  serves  as  a  proxy  measure  of  contract  success  or  failure  by  using  the 
ratings  given  to  each  contract  once  complete.  CPARS  ratings  are  given  in  Likert-style 
responses  where  l=Unsatisfactory,  2=Marginal,  3=Satisfactory,  4=Very  Good,  and 
5=Excellent.  In  this  case,  we  used  a  combined  CPARS  score  as  the  second  dependent 
variable.  Specifically,  we  searched  for  differences  in  CPARS  ratings  between  LPTA  and 
Trade-Off  acquisition  strategies  by  examining  a  within-contract  average  of  CPARS  Cost 
ratings,  CPARS  Quality  ratings,  CPARS  Schedule  ratings,  and  CPARS  Business 
Relationship  ratings. 

Our  model  has  one  independent  variable,  or  IV.  Independent  variables  are  those 
that  can  be  manipulated  by  the  researcher  (or  user)  to  evoke  a  change  in  the  outcome,  or 
DV.  In  this  case,  the  IV  concerns  the  contract  methodology  used  for  the  contract:  lowest 
price  technically  acceptable  (LPTA)  or  Trade-Off  (TO).  Contracting  methodology  is  a 
choice  made  by  the  “user”  (the  integrated  product  team,  which  includes  the  Contracting 
Officer),  hence  it  is  considered  an  IV.  Our  IV  is  labeled  LPTATO  and  it  is  a  binary 
variable  where  0=LPTA  and  l=Trade-Off. 

Finally,  there  are  four  covariate  variables.  Covariates  are  secondary  variables  that 
can  also  affect  the  relationship  of  primary  interest:  the  relationship  between  the  IV  and 


1  EVM  data  was  not  available  for  the  contracts  that  were  reviewed. 
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the  DV.  Specifically,  covariates  are  variables  other  than  the  independent  variable 
that  potentially  affect  the  outcome  variable,  or  DV.  In  this  case,  the  covariates  are  (1) 
number  of  evaluation  factors  (NUMEVALF  ACTORS),  (2)  number  of  reviews 
(NUMREVIEWS),  (3)  number  of  offers  (NUMOFFERS),  and  (4)  contract  dollar  value 
(VALUE).”  All  are  continuous  variables.  Theoretically,  each  of  the  first  three  covariates 
could  affect  the  PALT  positively  (e.g.,  a  larger  number  of  any  of  the  covariates  could 
increase  the  PALT,  or  time  to  contract).  Each  of  the  first  three  covariates  may  also  have  a 
positive  effect  on  CPARS  rating.  One  could  argue  that  a  higher  number  of  each  of  the 
covariates  serves  to  improve  the  chances  of  selecting  the  “right”  or  “best”  contractor, 
ultimately  improving  contract  perfonnance  (as  measured  by  CPARS  rating).  In  this  case, 
we  hope  to  parcel  out  the  effects  of  the  covariates  in  order  to  more  clearly  see  the  effect 
the  contracting  methodology  (LPTATO)  has  on  the  outcome  variables  (PALT  and 
CPARS  ratings).  It  is  worth  noting  that  not  all  of  our  covariates  passed  assumption 
testing,  therefore  not  all  were  used.  More  details  are  provided  in  Chapter  IV. 

E.  SUMMARY 

This  chapter  discussed  the  methodology  used  to  locate,  collect,  and  analyze  data 
in  order  to  answer  the  research  questions.  This  included  a  description  of  the  data  source, 
the  types  of  data  accessed,  and  the  data  collection  process.  The  last  section  described  the 
analysis  technique  that  was  used  to  quantify  the  relationship  (if  any)  between  source 
selection  method  and  contract  outcomes  such  as  CPARS  ratings  and  PALT.  The  next 
chapter  will  discuss  the  data  analysis  results. 


-  One  might  assume  that  NUMREVIEWS  and  VALUE  are  highly  correlated,  given  the  fact  that 
contract  dollar  value  often  determines  the  number  and  level  of  reviews  through  which  a  solicitation  and 
contract  must  pass.  However,  for  this  data,  we  found  they  were  not  highly  correlated  (r=.03),  thus  we  chose 
to  keep  both  as  covariates. 
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IV.  DATA  ANALYSIS 


This  chapter  describes  the  data  analysis  and  results  using  multivariate  and 
univariate  analysis  of  covariance  techniques  to  determine  if  there  are  differences  in 
contract  outcomes  based  on  source  selection  methodology.  The  analysis  is  further 
supported  by  a  discussion  of  the  descriptive  statistics  for  each  variable,  data  issues,  and 
assumption  testing. 

A.  DESCRIPTIVE  STATISTICS 

Basic  descriptive  statistics  for  each  variable  are  shown  in  Table  1.  The  table 
presents  three  figures  for  each  variable:  (1)  the  total  for  all  the  data,  (2)  the  total  for 
LPTA  contracts,  and  (3)  the  total  for  tradeoff  contracts. 


Table  1.  Descriptive  Statistics 


Variable 

Obs 

Mean 

SD 

Min 

Max 

PALT 

(days) 

23 

317.5652 

214.7235 

50 

878 

ALL  DATA 

10 

203.1 

141.6054 

50 

482 

LPTA 

13 

405.6154 

223.9898 

72 

878 

TRADEOFF 

Average 

CPARS 

Rating 

18 

3.766667 

.639623 

3 

5 

ALL  DATA 

5 

3.34 

.527257 

3 

4.2 

LPTA 

13 

3.930769 

.6179733 

3 

5 

TRADEOFF 

Number  of 
Evaluation 
Factors 

24 

2.666667 

.5646597 

2 

4 

ALL  DATA 

10 

2.1 

.3162278 

2 

3 

LPTA 

14 

3.071429 

.2672612 

3 

4 

TRADEOFF 

Number  of 
Reviews 

22 

3.909091 

3.66332 

1 

18 

ALL  DATA 

9 

2.555556 

2.068279 

1 

7 

LPTA 

13 

4.846154 

4.278749 

1 

18 

TRADEOFF 

Number  of 
Offers 

24 

6.833333 

5.329709 

2 

23 

ALL  DATA 

10 

8.1 

7.23341 

2 

23 

LPTA 

14 

5.928571 

3.452185 

2 

11 

TRADEOFF 

Contract 

Dollar 

Value 

24 

$268,000,000 

$595,000,000 

$1,185,396 

$2,300,000,000 

ALL  DATA 

10 

$3,947,115 

$2,218,574 

$1,185,396 

$6,500,000 

LPTA 

14 

$456,000,000 

$732,000,000 

$1,271,044 

$2,300,000,000 

TRADEOFF 
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B.  DATA  ISSUES 

With  only  24  cases,  the  sample  size  is  small,  and  the  distribution  of  cases  by 
source  selection  method  is  unbalanced.  Power  calculations  suggest  the  need  for  14  cases 
for  each  source  selection  methodology  (i.e.,  14  LPTA  cases  and  14  tradeoff  cases)  in 
order  to  achieve  adequate  power  (a  =  .05,  P  =  .80).  The  data  are  unbalanced  with  respect 
to  the  number  of  cases  for  each  source  selection  methodology.  There  are  10  LPTA  cases 
and  14  tradeoff  cases.  This  unbalanced  design  can  cause  ambiguity  about  the  mean  as  the 
intercept  and  make  assignment  of  sums  of  squares  more  difficult.  There  are,  however, 
solutions  to  these  issues.  A  weighted  mean  can  be  used  in  place  of  the  grand  mean  and 
the  STATA  software  automatically  handles  the  assignment  of  the  sums  of  squares.  Thus, 
we  proceeded  with  our  analysis  despite  these  issues. 

C.  ANALYSIS 

Because  our  intent  is  to  analyze  differences  in  contract  outcomes  (PALT  and 
CPARS  ratings)  based  on  source  selection  methodology  (LPTA  or  tradeoff),  a  group 
comparison  statistical  methodology  is  necessary.  In  other  words,  the  source  selection 
methods  are  divided  into  two  groups  (LPTA  and  tradeoff),  and  we  seek  to  find  if  there 
are  differences  in  contract  outcomes  (PALT  and  CPARS  ratings)  by  group. 

We  used  a  technique  called  multivariate  analysis  of  covariance,  or  MANCOVA, 
to  assess  group  differences.  MANCOVA  essentially  creates  a  new  dependent  variable 
using  the  information  from  the  given  dependent  variables  (PALT  and  CPARS  rating). 
This  new  dependent  variable  is  created  in  a  way  that  maximizes  differences  between  the 
grouping  variable  (LPTA  or  tradeoff  source  selections).  MANCOVA  addresses  the 
following  questions:  Are  mean  differences  among  the  groups  on  a  combination  of 
DVs  (after  adjusting  for  co variate  effects)  likely  to  have  occurred  by  chance?  Taken 
from  another  angle,  is  there  a  significant  difference  between  the  mean  value  for  PALT 
and  CPARS  rating  in  the  LPTA  acquisitions  versus  the  mean  value  for  PALT  and 


3 


The  grand  mean  would  be  the  intercept  in  a  balanced  design. 
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CPARS  ratings  in  the  tradeoff  acquisitions,  once  the  effects  of  the  covariates 
(NUMEVALFACTORS,  NUMREVIEWS,  NUMOFFERS,  VALUE)  have  been  parceled 
out?  MANCOVA  examines  the  relationships  between  the  dependent  variables  (PALT 
and  CPARS  ratings)  and  the  independent  variable  (choice  of  LPTA  or  tradeoff 
methodology)  while  taking  into  account  the  effects  the  covariates  might  have  on  the 
dependent  variables. 

If  differences  in  outcomes  are  found  using  MANCOVA,  researchers  typically 
choose  to  dig  deeper  into  the  differences  using  a  univariate  technique  called  analysis  of 
covariance  (ANCOVA).  ANCOVA  essentially  performs  the  same  group  differences 
analysis  as  a  MANCOVA,  however  because  it  is  univariate  in  nature,  this  method 
assesses  one  DV  at  a  time  (PALT  or  CPARS,  rather  than  PALT  and  CPARS).  This 
method  helps  researchers  isolate  where  the  difference  is  occurring  (which  IVs  or 
covariates  are  affecting  which  DVs).  We  use  both  MANCOVA  and  ANCOVA  in  this 
research. 

D.  ASSUMPTION  TESTING 

Before  conducting  the  MANCOVA,  certain  assumptions  about  the  data  were 
tested.  First,  we  assessed  multivariate  nonnality  by  examining  density  graphs, 
determining  multivariate  skewness  and  kurtosis,  and  by  examining  the  Doornik-Hansen 
test  for  multivariate  normality  (Doomik  &  Hansen,  2008).  For  the  DVs,  both  PALT  and 
CPARS  were  deemed  to  be  non-normal.  Both  variables  were  normalized  via  a 
logarithmic  transformation.  All  of  the  covariates  (NUMEVALF  ACTORS, 
NUMREVIEWS,  NUMOFFERS,  VALUE)  also  required  a  logarithmic  transfonnation, 
after  which  all  except  NUMEVALFACTORS  were  deemed  to  be  multivariate  normal. 
We  chose  to  drop  the  ill-behaved  NUMEVALFACTORS  covariate  from  all  subsequent 
analyses,  as  “[cjovariates  are  often  included  as  a  convenience  in  reducing  error,  but  it  is 
hardly  a  convenience  if  it  reduces  power”  (Tabachnick  &  Fidell,  2007,  p.  25 1). 

Second,  we  searched  for  outliers  using  Mahalanobis’  Distance.  We  found  no 
significantly  influential  outliers  in  our  data. 
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Third,  we  assessed  linearity  by  examining  scatter  plots  of  (1)  the  paired  DVs,  (2) 
all  pairs  of  covariates,  and  (3)  all  pairs  of  DV-covariate  combinations  for  each  source 
selection  method  (LPTA  and  tradeoff,  a  total  of  20  plots).  The  plots  revealed  that  all 
relationships  were  linear  in  nature. 

Fourth,  we  assessed  homogeneity  of  regression  by  performing  an  analysis  of 
variance  (ANOVA,  which  is  practically  the  same  as  an  ANCOVA  but  does  not  contain  a 
covariate  variable)  that  included  the  independent  variable  (LPTA  or  tradeoff),  each  of  the 
covariates  (NUMREVIEWS,  NUMOFFERS,  VALUE),  and  the  interaction  between  the 
independent  variable  and  the  covariate.  When  the  interaction  terms  are  not  significant,  the 
relationship  between  the  dependent  variables  (PALT  and  CPARS  ratings)  and  the 
covariates  is  the  same  at  both  levels  of  the  independent  variable  (LPTA  or  tradeoff), 
upholding  the  assumption  of  homogeneity  of  regression.  With  our  data,  significant 
interactions  were  found  between  (1)  PALT  and  VALUE,  (2)  CPARS  and  VALUE,  and 
(3)  CPARS  and  NUMREVIEWS.  Because  VALUE  failed  the  test  of  homogeneity  of 
regression  for  both  DVs,  it  was  completely  dropped  from  further  analysis.  Further, 
NUMREVIEWS  could  not  be  included  in  the  MANCOVA,  as  it  failed  the  homogeneity 
of  regression  test  for  the  CPARS  outcome  variable.  Thus,  NUMOFFERS  is  the  only 
covariate  that  is  available  for  the  MANCOVA.  It  is  important  to  note  that  we  use 
NUMREVIEWS  in  an  ANCOVA  (univariate)  assessment  of  the  relationship  between 
source  selection  methodology  (LPTA  or  tradeoff)  and  PALT,  as  NUMREVIEWS  did  not 
fail  the  homogeneity  of  regression  test  for  the  PALT  outcome  variable. 

Fifth,  we  checked  for  multicollinearity  by  assessing  the  pooled  within  cell 
tolerance  for  each  DV  (PALT  and  CPARS).  Multicollinearity  is  not  an  issue  for  our  data. 

Finally,  we  checked  for  homogeneity  of  covariance  matrices  between  groups  (i.e., 
sphericity)  using  the  multivariate  test  of  means  provided  in  STATA  12.1.  This  test  checks 
whether  or  not  population  variances  and  covariances  of  both  dependent  variables  (PALT 
and  CPARS)  are  equal  for  each  of  the  independent  variable  groups  (LPTA  and  tradeoff). 
The  results  showed  that  all  grouping  cells  were  homogenous. 
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With  all  assumptions  tested,  we  performed  the  MANCOVA  (with  only 
NUMOFFERS  as  a  covariate)  and  subsequent  ANCOVA.  The  results  are  provided 
below. 

E.  RESULTS 

1.  MANCOVA 

The  results  show  that  source  selection  methodology  (LPTA  or  Tradeoff)  produced 
significantly  different  contracting  outcomes  at  the  p<.05  level.  See  Table  2.  Specifically, 
we  found  that  tradeoff  source  selections  produce  higher  CPARS  ratings  (23%  higher 
CPARS  score). 

Breaking  down  the  results  further,  we  found  that  the  NUMOFFERS  submitted 
also  serves  to  increase  CPARS  score  (12%  higher  score  for  each  percentage  increase  in 
NUMOFFERS,  p<.10).  Table  2  shows  the  results  of  this  analysis. 


Table  2.  MANCOVA  Results— DVs:  PALT  and  CPARS  Ratings 


MANCOVA  Results 

DVs:  PALT  and  CPARS  Rating 

Coeff 

SE 

t 

P>|t| 

95%  Cl 

PALT 

LPTATO 

.4635959 

.3353186 

1.38 

.188 

-.255591 

1.182783 

#  Offers 

.0856102 

.2357787 

0.36 

0.722 

-.4200848 

.5913052 

Constant 

5.336645 

.5807349 

9.19 

0.000 

4.091093 

6.582198 

CPARS 

LPTATO 

.2347992** 

.0885981 

2.65 

0.019 

.0447753 

.4248231 

#  Offers 

.1194312* 

.0622976 

1.92 

0.076 

-.0141838 

-.2530462 

Constant 

.9350972 

.1534421 

6.09 

0.000 

.6059967 

1.264198 

*  p<.10  **  p<.05  ***  p<.01 

Number  of  Observations  =17 

Root  Mean  Squared  Error:  PALT  =  .57  CPARS  =  .15 

R2:  PALT  =  .1231  CPARS  =  .3585 

2.  ANCOVA 

Given  that  two  covariates  (NUMRE VIEWS  and  NUMOFFERS)  passed  the 
homogeneity  of  regression  test  for  the  outcome  variable  PALT,  we  decided  to  conduct  a 
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post-hoc  analysis  to  see  if  there  was  a  significant  difference  in  PALT  based  on  source 
selection  methodology  (LPTA  or  tradeoff).  Again,  because  we  are  only  using  one  DV 
(PALT),  we  are  using  the  univariate  method  (ANCOVA),  not  the  multivariate  method 
(MANCOVA).  The  results  of  the  ANCOVA  show  that  no  differences  in  PALT  exist 
based  on  source  selection  method.  However,  the  covariate  NUMREVIEWS  significantly 
increases  PALT  (65%  increase  when  using  the  tradeoff  method,  p<.01).  This  result  is 
intuitive,  as  contract  reviews  add  time  to  the  procurement  process  and  increasing  the 
number  of  reviews  will  increase  PALT.  Typically,  contract  reviews  are  conducted 
consecutively  and  require  approval  before  proceeding  to  the  next  step  in  the  procurement 
process  or  to  the  next  review.  Table  3  shows  the  results  of  this  analysis. 


Table  3 .  ANCOVA  Results— D Vs:  PALT 


ANCOVA  Results 

DVs:  PALT 

Coeff 

SE 

t 

P>|t| 

95%  Cl 

PALT 

#Reviews 

.6543624*** 

.1767935 

3.70 

0.002 

.2829331 

1.025792 

#  Offers 

.1433843 

.1771919 

0.81 

0.429 

-.2288821 

.5156508 

LPTATO 

.1802472 

.2665075 

0.68 

0.507 

-.3796642 

.7401586 

Constant 

4.51685 

.3458353 

13.06 

0.000 

3.790277 

5.243422 

*  p<.10  **  p<.05  ***  p<.01 

Number  of  Observations  =  22 

Root  Mean  Squared  Error  =  .55236 

R2=  0.5551 

Adjusted  R2  =  0.48 10 

Perhaps  more  telling  are  the  relationships  between  the  variables  themselves.  The 
correlations  are  given  in  Table  4.  The  lower  triangle  presents  correlations  of  LPTA 
contracts,  and  the  upper  triangle  presents  correlations  of  tradeoff  contracts. 
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Table  4.  Correlation  Data 


^xTradeoff 

LPTA  X. 

PALT 

CPARS 

#  Evaluation 
Factors 

#  Reviews 

#  Offers 

Value 

PALT 

1.00 

-0.6223 

0.6692 

0.7473 

-0.0580 

-0.0446 

CPARS 

0.6393 

1.00 

-0.4423 

-0.4888 

0.5523 

-0.3941 

# 

Evaluation 

Factors 

0.2671 

0.9118 

1.00 

0.9392 

-0.0940 

0.0388 

#  Reviews 

0.7003 

0.9966 

0.8750 

1.00 

-0.0650 

-0.1648 

#  Offers 

0.8953 

0.2298 

-0.1901 

0.3089 

1.00 

-0.6449 

Value 

0.9394 

0.8641 

0.5812 

0.9025 

0.6884 

1.00 

Variables  with  a  correlation  coefficient  of  +0.70  or  greater  indicates  a  strong 
positive  linear  relationship  reaching  perfect  correlation  at  +1.  Variables  with  a  correlation 
coefficient  of  -0.70  or  less  indicates  a  strong  negative  linear  relationship  reaching  perfect 
correlation  at  -1.  A  coefficient  of  0  indicates  that  no  relationship  exists. 

a.  LPTA  Correlations 

CPARS  and  PALT  have  a  moderately  positive  relationship  at  .6393.  This  suggests 
that  taking  additional  time  in  the  acquisition  process  (e.g.,  to  craft  the  best  solicitation 
possible,  to  ensure  a  thorough  review  of  offers,  to  make  a  well-informed  source  selection 
decision,  etc.)  under  an  LPTA  method  leads  to  the  selection  of  better  performing 
contractors.  Further,  the  almost  perfect  positive  correlation  between  NUMREVIEWS  and 
CPARS  at  .9966  indicates  that  more  reviews  lead  to  the  selection  of  better  perfonning 
contractors. 

NUMEVALFACTORS  and  CPARS  have  a  strong  positive  relationship  at  0.91 18. 
This  is  an  interesting  correlation  because  it  suggests  that  increasing  the  number  of 
evaluation  factors  leads  to  the  selection  of  better-performing  contractors.  Factors  under 
an  LPTA  method  are  evaluated  based  on  pass  or  fail  basis.  The  more  factors  there  are,  the 
more  aspects  of  the  technical  and  non-price  areas  of  the  requirement  that  are  evaluated; 
therefore,  this  relationship  suggests  better  performing  contractors  are  able  to  pass  more 
hurdles  in  terms  of  evaluation  factors  than  less  qualified  contractors. 
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NUMREVIEWS  and  PALT  have  a  strong  positive  relationship  of  .7003.  This  is 
an  expected  relationship,  because  each  review  takes  time  and  all  reviews  are  typically 
consecutive.  Even  though  requirements  using  LPTA  are  less  complex  than  tradeoff, 
additional  reviews  will  increase  PALT. 

NUMOFFERS  and  PALT  have  a  strong  positive  relationship  of  0.8953.  This 
result  is  also  intuitive.  The  more  proposals  a  source  selection  evaluation  teams  has  to 
evaluate,  the  longer  the  PALT.  Additionally,  more  offers  could  lead  to  increased 
exchanges  with  offerors  through  clarifications,  communications,  and  additional  rounds  of 
discussions  and  resulting  proposal  changes,  all  of  which  serve  to  increase  PALT. 

VALUE  and  PALT  have  a  strong  positive  relationship  of  0.9394.  This  is  likely 
the  result  of  the  fact  that  higher  value  contracts  typically  require  more  reviews  because 
many  regulatory  procedures  are  based  on,  or  triggered  by,  dollar  thresholds.  The  strong 
positive  relationship  of  .9025  between  Value  and  the  number  of  reviews  also  supports 
this  relationship. 

b.  Tradeoff  Correlations 

Variables  for  tradeoff  contracts  did  not  exhibit  similar  correlation  patterns  as 
LPTA  contracts.  Overall,  tradeoff  variables  exhibited  more  negative  relationships. 

CPARS  and  PALT  have  moderately  negative  relationships  of  -0.6223.  It  is  not 
clear  why  an  increase  in  PALT  would  lead  to  a  lower  CPARS  rating  and  vice-versa. 
Additionally,  NUMREVIEWS  and  CPARS  also  has  a  negative  relationship  (r  =  -.4888). 
These  results  are  opposite  of  the  results  observed  under  the  LPTA  method  for  the  same 
relationships.  It  is  unclear  why  this  difference  exists. 

NUMEVALF ACTORS  and  PALT  have  moderately  positive  relationships  of 
0.6692.  This  is  likely  an  indication  that  increasing  the  number  of  evaluation  factors 
increases  the  amount  of  time  it  takes  the  source  selection  team  to  complete  proposal 
evaluations,  thus  increasing  PALT.  NUMREVIEWS  also  had  a  strong  positive 
relationship  with  PALT  of  0.7473,  which  is  similar  to  the  correlation  for  LPTA  contracts. 


44 


This  is  expected  because  contract  reviews  typically  increase  the  PALT  regardless  of  the 
chosen  source  selection  method. 

Surprisingly,  NUMEVALFACTORS  and  CPARS  and  NUMREVIEWS  and 
CPARS  have  low  negative  relationships  of  -0.4423  and  -0.4888,  respectively.  This  is 
contrary  to  the  strong  positive  relationship  between  these  variables  for  LPTA  contracts. 
These  results  are  counterintuitive,  as  one  would  theoretically  expect  a  higher  number  of 
evaluation  factors  and  reviews  to  yield  a  better-perfonning  contractor.  More  data  is 
needed  to  detennine  the  stability  and  significance  of  these  negative  relationships. 

NUMOFFERS  and  CPARS  have  moderately  positive  relationships  of  0.5523. 
This  is  intuitive  based  on  the  expectation  that  more  offers  provides  the  SSA  with  more 
options  to  choose  from,  which  leads  to  a  better  chance  of  selecting  a  high-performing 
contractor. 

Our  analysis  presented  some  unexpected  results  for  the  tradeoff  correlations. 
More  data  and  further  analysis  are  necessary  to  check  the  stability  of  these  results. 

F.  SUMMARY 

This  chapter  discussed  the  data  analysis  and  results  using  MANCOVA  and 
ANCOVA  techniques.  It  also  included  some  post-hoc  analyses  which  looked  closer  at  the 
relationships  between  covariates  and  contract  outcomes.  An  examination  of  the 
relationships  between  some  of  the  covariates  and  contract  outcomes  identified  areas  for 
further  research.  The  next  chapter  will  provide  our  conclusions  and  recommendations 
based  on  our  findings. 
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V.  CONCLUSIONS 


This  chapter  summarizes  the  purpose,  analysis,  and  findings  of  this  research,  and 
answers  the  research  questions  posed  in  Chapter  I.  The  chapter  concludes  with 
recommendations  for  further  research. 

A.  SUMMARY 

As  stewards  of  taxpayer  dollars,  government  acquisition  teams  have  an  obligation 
to  provide  the  best  value  for  products  and  services  required  by  their  customers.  While 
conducting  a  successful  source  selection  is  a  critical  part  of  the  acquisition  process,  the 
ability  to  achieve  best  value  is  affected  by  the  activities  performed  during  all  the  phases 
of  the  federal  contract  management  process.  Within  the  DOD,  LPTA  and  tradeoff  are  the 
two  most  widely  used  source  selection  methods  on  the  best  value  continuum.  The  trend 
for  utilizing  LPTA  in  the  DOD  increased  over  the  past  few  years  along  with  sharp 
criticism  of  the  method  by  the  defense  industry.  Contractors  maintain  that  the  DOD  is 
using  the  LPTA  method  inappropriately  and  that  it  robs  the  government  of  high  quality 
products  and  services  and  innovative  solutions.  The  GAO  disputes  these  claims,  offering 
report  findings  that  assert  the  DOD  is  using  the  method  appropriately.  Currently,  there  is 
no  empirical  evidence  that  links  the  utilization  of  the  LPTA  source  selection  method  to 
poor  contactor  perfonnance  outcomes  or  the  use  of  the  tradeoff  method  to  good 
performance  outcomes.  The  objective  of  this  research  was  to  determine  whether  the 
source  selection  method  employed  by  the  Air  Force,  either  LPTA  or  tradeoff,  affects  the 
time-to-contract  metric  (PALT)  and/or  the  contractor’s  performance  (as  measured  by 
CPARS). 

B.  CONCLUSION 

Based  on  our  results,  we  can  now  answer  the  research  questions  that  were 
presented  in  Chapter  I: 

1 .  What  is  the  relationship  between  a  particular  source  selection  method  and 
the  resulting  performance  outcome  of  the  contractor? 
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For  the  first  question,  our  analysis  supports  that  a  relationship  exists  between  a 
particular  source  selection  method  (LPTA  or  tradeoff)  and  the  resulting  contractor 
performance  outcome.  We  found  that  source  selections  conducted  under  the  tradeoff 
method  produced  higher  perfonnance  outcomes  as  measured  by  CPARS.  Additionally, 
we  found  that  there  is  a  positive  relationship  between  the  number  of  offers  submitted  and 
the  perfonnance  outcome  of  the  contractor  as  measured  by  CPARS.  In  other  words,  the 
more  competitive  the  acquisition,  the  better  the  performance  outcome  of  the  selected 
contractor. 

2.  What  is  the  relationship  between  a  particular  source  selection  method  and 
the  PALT? 

i.  If  there  is  a  positive  relationship,  can  the  amount  of  additional 
resources  utilized  for  that  particular  source  selection  method  be 
quantified  in  terms  of  man-hours  or  in  dollars? 

For  the  second  question  and  its  first  sub-question,  our  analysis  does  not  support  a 
relationship  between  a  particular  source  selection  method  and  the  PALT.  Nonetheless, 
there  are  covariate  factors  that  can  affect  the  PALT.  For  example,  the  number  of  reviews 
positively  impacts  the  PALT  under  both  LPTA  and  tradeoff  source  selection  methods, 
which  means  that  the  more  reviews  there  are,  the  longer  time  it  takes  to  get  to  award.  As 
a  second  example,  the  number  of  offers  affects  the  PALT  positively  when  using  LPTA 
but  slightly  negatively  when  using  tradeoff.  More  specifically,  the  greater  the  number  of 
offers  to  evaluate  under  an  LPTA  source  selection,  the  longer  it  takes  to  get  to  award,  yet 
greater  numbers  of  offers  under  a  tradeoff  source  selection  slightly  reduces  time  to  award. 
Clearly,  this  ambiguous  finding  requires  further  research  to  disentangle. 

ii.  Does  the  resulting  perfonnance  outcome  merit  the  use  of 
that  particular  source  selection  method? 

Since  our  research  seems  to  support  the  fact  that  a  contractor  selected  through  the 
tradeoff  method  has  higher  performance  outcomes  than  under  LPTA,  the  answer  to  this 
sub-question  is  yes,  the  tradeoff  method  produces  better  perfonnance,  based  on  CPARS 
scores.  Moreover,  since  PALT  is  not  affected  by  the  source  selection  method,  there  is  no 
reason  to  shy  away  from  the  use  of  the  tradeoff  method  to  achieve  better  contractor 
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performance.  Clearly,  these  results  are  specific  to  our  data,  and  the  collection  of 
additional  data  is  necessary  to  confirm  these  findings. 


C.  RECOMMENDATIONS  FOR  FURTHER  RESEARCH 

This  research  project  produced  some  interesting  results  with  data  collected  from 
the  AFLCMC,  which  is  only  one  center  within  an  Air  Force  MAJCOM  (AFMC)  known 
for  its  prominence  in  acquisitions.  We  recommend  that  additional  data  be  collected  and 
analyzed  to  confirm  the  findings  of  our  research.  Furthermore,  due  to  time  and  manpower 
constraints,  and  the  fact  that  the  AFLCMC  customarily  engages  in  sole  source  contracts, 
it  was  difficult  for  us  to  collect  a  well-balanced  and  statistically  powerful  amount  of  data. 
We  recommend  repeating  this  research  in  its  entirety  with  a  larger  sample  size  and  data 
representing  all  the  organizations  in  AFMC. 

Since  the  AFMC  consists  of  mostly  systems  level  procurements,  it  would  be 
beneficial  to  repeat  this  research  across  the  Air  Force  and  other  agencies  to  assess  the 
outcomes  for  different  types  of  requirements  across  the  DOD.  The  results  may  vary 
according  to  the  type  of  requirement  (i.e.,  construction,  services,  and  commodities). 
Understanding  these  relationships  across  the  various  requirement  types  can  help 
acquisition  teams  plan  their  acquisition  strategy  in  a  way  that  cuts  waste  and  streamlines 
the  acquisition  process  while  still  achieving  best  value. 

Our  final  recommendation  is  to  devise  a  research  project  focused  on  quantifying 
the  amount  or  level  of  resources  consumed  during  the  course  of  a  procurement  process 
from  beginning  to  end.  We  expected  to  find  a  significant  relationship  between  source 
selection  method  and  PALT  due  to  the  fact  that  average  PALT  for  tradeoff  is  double  the 
average  PALT  for  LPT  A,  however  our  results  did  not  support  such  a  relationship. 
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APPENDIX  A.  BUSINESS  SECTOR,  DOLLAR  THRESHOLD 
AND  REVIEWING  OFFICIAL  (FROM  CPARS,  2015) 


Dollar  Threshold^ 

2 

Reviewing  Official 

Civilian  Agencies  (excludes 
DOD): 

Systems  and  Non-Systems 

Simplified 

Acquisition 

Threshold 

One  level  above  the  contracting 
officer,  as  determined  by  department 
or  agency  policy 

Architect-Engineer 

>$30,000; 

All  Terminations 
for  Default 

One  level  above  the  contracting 
officer,  as  determined  by  department 
or  agency  policy 

Construction 

>$650,000; 

All  Terminations 
for  Default 

One  level  above  the  contracting 
officer,  as  determined  by  department 
or  agency  policy 

DOD  Services  and  Agencies: 
Systems  (includes  new 
development  and  major 

>$5,000,000 

One  level  above  the  PMJ 

Non-Systems 

Operations  Support 

>$5,000,0004 

One  level  above  the  AO 

Services 

>$1,000,000 

One  level  above  the  AO 

Information  Technology 

>$1,000,000 

One  level  above  the  AO 

Ship  Repair  and  Overhaul 

>$500,000 

One  level  above  the  AO 

Architect-Engineer 

>$30,000; 

All  Terminations 
for  Default 

One  level  above  the  AO 

Construction 

>$650,000; 

All  Terminations 
for  Default 

One  level  above  the  AO 

The  contract/order  thresholds  for  CPARS  collection  (see  FAR  42. 1 502)  apply  to  the  “aggregate”  value  of 
contracts/orders;  that  is,  if  a  contract’s/order’s  original  award  value  were  less  than  the  applicable  threshold  but 
subsequently  the  contract/order  was  modified  and  the  new  value  is  greater  than  the  threshold,  then  evaluations  are 
required  to  be  made,  starting  with  the  first  anniversary  that  the  contract’s/order’s  face  value  exceeded  the  threshold. 
If  the  total  contract/order  value  including  unexercised  options  and  orders  (for  IDIQ  contracts,  total  estimated  value 
of  unexercised  options  and  orders)  is  expected  to  exceed  the  collection  threshold,  initiate  the  collection  process  at 
the  start  of  the  contract/order.  Buying  activities  may  choose  to  collect  performance  evaluations  for  awards  below 
these  thresholds. 

2 

Only  required  when  the  contractor  indicates  non-concurrence  with  the  CPARS  or  if  otherwise  requested 
by  the  contractor  during  the  60-calendar  day  comment  period. 

3 

(Or  equivalent  individual)  responsible  for  program,  project  or  task/job  order  execution. 

4 

For  contracts/orders  under  the  reporting  thresholds,  buying  activities  should  continue  to  accumulate 
contractor  performance  data  from  existing  management  information  systems,  which  already  capture  data 
on  timeliness  of  delivery  and  quality  of  product  or  service  (an  example  of  a  performance  infonnation 
collection  systems  is  the  Past  Performance  Information  Retrieval  System— Statistical  Reporting  Module) 
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APPENDIX  B.  EVM  REQUIREMENTS  (FROM  OUSD[AT&L],  2015) 


REQUIREMENTS 

WHEN  REQUIRED1 

SOURCE 

For  Cost/Incentive  Contracts2  >  $50  Million3 

Compliance  with  EVM 
system  guidelines  in  ANSI/EIA- 

At  contract  award  and  throughout 
contract  performance 

Part  7  of  Office  of 
Management  and 

Budget  (OMB)  Circular 

A-1 1  (Reference  ©) 

This  instruction 

EVM  system  formally  validated 
and  accepted  by  cognizant  contracting 

At  contract  award  and  throughout 
contract  performance 

Integrated  Program 

Management  Report  (IPMR)  (Dl- 

Monthly 

Integrated  Baseline  Reviews 

Within  1 80  calendar  days  after  contract 
award,  exercise  of  options,  and  major 

For  Cost/Incentive  Contracts2  >  $20  Million3  but  <  $50  Million3 

Compliance  with  EVM 
system  guidelines  in  ANSI/EIA- 
7484 

At  contract  award  and  throughout 
contract  performance 

Part  7  of  OMB  Circular  A- 
11  This  instruction 

-  IPMR  (DI-MGMT-818615) 
(tailoring  of  formats  recommended) 

Monthly 

Integrated  Baseline  Reviews 

For  Cost/Incentive  Contracts2  < 

$20  Million3 

Within  1 80  calendar  days  after  contract 
award,  exercise  of  options,  and  major 

At  the  discretion  of  the  Program  Manager 
based  on  cost-benefit  analysis 

Part  7  of  OMB  Circular  A- 
11  This  instruction 

-  IPMR,  Format  6  (DI-MGMT- 

For  Firm  Fixed-Price 

Contracts2  regardless  of 

At  the  discretion  of  the  Program  Manager 
hased  on  Government  renuirements 

Limited  Use-will  be  approved  by  the  MDA 
based  on  a  business  case  analysis 

Part  7  of  OMB  Circular  A- 
11  This  instruction 

-  IPMR,  Format  6  (DI-MGMT- 

At  the  discretion  of  the  Program  Manager 
based  on  Government  reauirements 

Notes: 

1 .  EVM  is  required,  as  outlined  in  the  table,  unless  the  EVM  requirement  has  been  waived  by  the  CAE  per  paragraph  6c  in  Enclosure 

2. 

2.  The  term,  “Contracts,”  includes  contracts,  subcontracts,  intra-government  work  agreements,  and  other  agreements.  For  Indefinite 
Delivery/Indefinite  Quantity  contracts,  EVM  will  be  applied  to  the  individual  task  orders  or  group  of  related  task  orders  in  accordance  with 
the  requirements  in  this  table.  “Incentive”  contracts  include  fixed-price  incentive.  EVM  is  required  for  Fixed-Price  Incentive  Fee 
development  and  integration  contracts  with  measurable  and  discrete  work  scope.  In  cases  where  the  work  scope  is  not  measurable 
and  discrete,  program  offices  should  follow  the  process  to  obtain  a  DFARS  deviation. 

3.  Application  thresholds  are  in  then-year  dollars. 

4.  ANSI/EIA-748  =  American  National  Standards  Institute  (ANSI)/Electronic  Industries  Alliance  (EIA)  Standard  748,  Earned 

Value  Management  Systems  (Reference  (au)). 

5.  DI-MGMT-81861  =  DID:  Integrated  Program  Management  Report  (Reference  (av)) 

a.  If  EVM  is  not  required  or  a  deviation  is  obtained,  the  IPMR  should  be  used  and  tailored  to  obtain  cost  and/or  schedule  reporting 
when  desired  by  the  Government.  For  example,  for  full  rate  production  contracts  where  EVM  is  not  applicable,  a  tailored  IPMR 
including  a  cost  report  showing  actuals  and  a  top-level  schedule  providing  delivery  dates  of  end  products  would  be  sufficient  for 
Government  management  and  oversight. 

b.  Flow-down  of  the  IPMR  DID  to  the  subcontractors  is  at  the  discretion  of  the  program  office. 
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